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Preface 


Revising this textbook has been a special challenge, for a very nice reason. So many 
people have read this book, and taught from it, and even loved it. The spirit of the book 
could never change. This text was written to help our teaching of linear algebra keep up 
with the enormous importance of this subject—which just continues to grow. 


One step was certainly possible and desirable—to add new problems. Teaching for all 
these years required hundreds of new exam questions (especially with quizzes going onto 
the web). I think you will approve of the extended choice of problems. The questions are 
still a mixture of explain and compute—the two complementary approaches to learning 
this beautiful subject. 


I personally believe that many more people need linear algebra than calculus. Isaac 
Newton might not agree! But he isn’t teaching mathematics in the 21st century (and 
maybe he wasn’t a great teacher, but we will give him the benefit of the doubt). Cer- 
tainly the laws of physics are well expressed by differential equations. Newton needed 
calculus—quite right. But the scope of science and engineering and management (and 
life) is now so much wider, and linear algebra has moved into a central place. 


May I say a little more, because many universities have not yet adjusted the balance 
toward linear algebra. Working with curved lines and curved surfaces, the first step is 
always to linearize. Replace the curve by its tangent line, fit the surface by a plane, 
and the problem becomes linear. The power of this subject comes when you have ten 
variables, or 1000 variables, instead of two. 


You might think I am exaggerating to use the word “beautiful” for a basic course 
in mathematics. Not at all. This subject begins with two vectors v and w, pointing in 
different directions. The key step is to take their linear combinations. We multiply to 
get 3v and 4w, and we add to get the particular combination 3v + 4w. That new vector 
is in the same plane as v and w. When we take all combinations, we are filling in the 
whole plane. If I draw v and w on this page, their combinations cv + dw fill the page 
(and beyond), but they don’t go up from the page. 


In the language of linear equations, I can solve cv + dw = b exactly when the vector 
b lies in the same plane as v and w. 


Matrices 
I will keep going a little more to convert combinations of three-dimensional vectors into 
linear algebra. If the vectors are v = (1,2,3) and w = (1,3,4), put them into the columns 


of a matrix: 


matrix = 


WN 
BR WD Re 


To find combinations of those columns, “multiply” the matrix by a vector (c,d): 


1 1 1 1 
Linear combinations cv + dw 2 3 a =c|2| +d {3 
3 4 3 4 


Those combinations fill a vector space. We call it the column space of the matrix. (For 
these two columns, that space is a plane.) To decide if b = (2,5,7) is on that plane, we 
have three components to get right. So we have three equations to solve: 


1 1 2 c+ d=2 
2453 4 = 15 means 2c+3d=5. 
3 4 7 3c+4d=7 


I leave the solution to you. The vector b = (2,5,7) does lie in the plane of v and w. 
If the 7 changes to any other number, then b won’t lie in the plane—it will not be a 
combination of v and w, and the three equations will have no solution. 

Now I can describe the first part of the book, about linear equations Ax = b. The 
matrix A has n columns and m rows. Linear algebra moves steadily to n vectors in m- 
dimensional space. We still want combinations of the columns (in the column space). 
We still get m equations to produce b (one for each row). Those equations may or may 
not have a solution. They always have a least-squares solution. 

The interplay of columns and rows is the heart of linear algebra. It’s not totally easy, 
but it’s not too hard. Here are four of the central ideas: 


1. The column space (all combinations of the columns). 

2. The row space (all combinations of the rows). 

3. The rank (the number of independent columns) (or rows). 
4. Elimination (the good way to find the rank of a matrix). 


I will stop here, so you can start the course. 
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Web Pages 


It may be helpful to mention the web pages connected to this book. So many messages 
come back with suggestions and encouragement, and I hope you will make free use 
of everything. You can directly access http://web.mit.edu/18.06, which is continually 
updated for the course that is taught every semester. Linear algebra is also on MIT’s 
OpenCourseWare site http://ocw.mit.edu, where 18.06 became exceptional by including 
videos of the lectures (which you definitely don’t have to watch...). Here is a part of 
what is available on the web: 


1. Lecture schedule and current homeworks and exams with solutions. 
2. The goals of the course, and conceptual questions. 

3. Interactive Java demos (audio is now included for eigenvalues). 

4. Linear Algebra Teaching Codes and MATLAB problems. 


5. Videos of the complete course (taught in a real classroom). 


The course page has become a valuable link to the class, and a resource for the students. 
I am very optimistic about the potential for graphics with sound. The bandwidth for 
voiceover is low, and FlashPlayer is freely available. This offers a quick review (with 
active experiment), and the full lectures can be downloaded. I hope professors and 
students worldwide will find these web pages helpful. My goal is to make this book as 
useful as possible with all the course material I can provide. 


Other Supporting Materials 


Student Solutions Manual 0-495-01325-0 The Student Solutions Manual provides 
solutions to the odd-numbered problems in the text. 


Instructor’s Solutions Manual 0-030-10588-4 The Instructor’s Solutions Man- 
ual has teaching notes for each chapter and solutions to all of the problems in the text. 


Structure of the Course 


The two fundamental problems are Ax = b and Ax = Ax for square matrices A. The first 
problem Ax = b has a solution when A has independent columns. The second problem 
Ax = Ax looks for independent eigenvectors. A crucial part of this course is to learn 
what “independence” means. 

I believe that most of us learn first from examples. You can see that 


iL k'2 
A= bes 2'3 does not have independent columns. 
13 4 


Vil 


Column 1 plus column 2 equals column 3. A wonderful theorem of linear algebra says 
that the three rows are not independent either. The third row must lie in the same plane 
as the first two rows. Some combination of rows | and 2 will produce row 3. You might 
find that combination quickly (I didn’t). In the end I had to use elimination to discover 
that the right combination uses 2 times row 2, minus row 1. 

Elimination is the simple and natural way to understand a matrix by producing a lot 
of zero entries. So the course starts there. But don’t stay there too long! You have to get 
from combinations of the rows, to independence of the rows, to “dimension of the row 
space.” That is a key goal, to see whole spaces of vectors: the row space and the column 
space and the nullspace. 

A further goal is to understand how the matrix acts. When A multiplies x it produces 
the new vector Ax. The whole space of vectors moves—it is “transformed” by A. Special 
transformations come from particular matrices, and those are the foundation stones of 
linear algebra: diagonal matrices, orthogonal matrices, triangular matrices, symmetric 
matrices. 

The eigenvalues of those matrices are special too. I think 2 by 2 matrices provide 
terrific examples of the information that eigenvalues A can give. Sections 5.1 and 5.2 
are worth careful reading, to see how Ax = Ax is useful. Here is a case in which small 
matrices allow tremendous insight. 

Overall, the beauty of linear algebra is seen in so many different ways: 


1. Visualization. Combinations of vectors. Spaces of vectors. Rotation and reflection 
and projection of vectors. Perpendicular vectors. Four fundamental subspaces. 


2. Abstraction. Independence of vectors. Basis and dimension of a vector space. 
Linear transformations. Singular value decomposition and the best basis. 


3. Computation. Elimination to produce zero entries. Gram-Schmidt to produce 
orthogonal vectors. Eigenvalues to solve differential and difference equations. 


4. Applications. Least-squares solution when Ax = b has too many equations. Dif- 
ference equations approximating differential equations. Markov probability matrices 
(the basis for Google!). Orthogonal eigenvectors as principal axes (and more...). 


To go further with those applications, may I mention the books published by Wellesley- 
Cambridge Press. They are all linear algebra in disguise, applied to signal processing 
and partial differential equations and scientific computing (and even GPS). If you look 
at http://www.wellesleycambridge.com, you will see part of the reason that linear algebra 
is so widely used. 

After this preface, the book will speak for itself. You will see the spirit right away. 
The emphasis is on understanding—T/ try to explain rather than to deduce. This is a 
book about real mathematics, not endless drill. In class, I am constantly working with 
examples to teach what students need. 
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Matrices and Gaussian Elimination 


1.1. Introduction 


This book begins with the central problem of linear algebra: solving linear equations. 

The most important ease, and the simplest, is when the number of unknowns equals the 

number of equations. We have n equations in n unknowns, starting with n = 2: 
Two equations 1x + 2y = 3 (1) 
Two unknowns 4x + 5y = 6. 

The unknowns are x and y. I want to describe two ways, elimination and determinants, 

to solve these equations. Certainly x and y are determined by the numbers 1, 2, 3, 4, 5, 

6. The question is how to use those six numbers to solve the system. 


1. Elimination Subtract 4 times the first equation from the second equation. This 
eliminates x from the second equation. and it leaves one equation for y: 


(equation 2) — 4(equation 1) — 3y= —-6. (2) 
Immediately we know y = 2. Then x comes from the first equation 1x + 2y = 3: 
Back-substitution Ix+2(2)=3 gives x=-l. (3) 


Proceeding carefully, we cheek that x and y also solve the second equation. This 
should work and it does: 4 times (x = —1) plus 5 times (y = 2) equals 6. 


2. Determinants The solution y = 2 depends completely on those six numbers in the 
equations. There most be a formula for y (and also x) It is a “ratio of determinants” 
and I hope you will allow me to write it down directly: 


Re 


3 

6] 1.6—3- 
mats = =2. (4) 

A|. eed SB 

5 


Re 
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That could seem a little mysterious, unless you already know about 2 by 2 determi- 
nants. They gave the same answer y = 2, coming from the same ratio of —6 to —3. 
If we stay with determinants (which we don’t plan to do), there will be a similar 
formula to compute the other unknown, x: 


Sie 
_ |6 5} 3-5-2-6 3 
12 1-5-2-4 —-3 
4 5 


=-1, (5) 


Let me compare those two approaches, looking ahead to real problems when n is 
much larger (n = 1000 is a very moderate size in scientific computing). The truth is that 
direct use of the determinant formula for 1000 equations would be a total disaster. It 
would use the million numbers on the left sides correctly, but not efficiently. We will 
find that formula (Cramer’s Rule) in Chapter 4, but we want a good method to solve 
1000 equations in Chapter 1. 

That good method is Gaussian Elimination. This is the algorithm that is constantly 
used to solve large systems of equations. From the examples in a textbook (n = 3 is 
close to the upper limit on the patience of the author and reader) too might not see much 
difference. Equations (2) and (4) used essentially the same steps to find y = 2. Certainly 
x came faster by the back-substitution in equation (3) than the ratio in (5). For larger 
n there is absolutely no question. Elimination wins (and this is even the best way to 
compute determinants). 

The idea of elimination is deceptively simple—you will master it after a few exam- 
ples. It will become the basis for half of this book, simplifying a matrix so that we can 
understand it. Together with the mechanics of the algorithm, we want to explain four 
deeper aspects in this chapter. They are: 


1. Linear equations lead to geometry of planes. It is not easy to visualize a nine- 
dimensional plane in ten-dimensional space. It is harder to see ten of those planes, 
intersecting at the solution to ten equations—but somehow this is almost possible. 
Our example has two lines in Figure 1.1, meeting at the point (x,y) = (—1,2). 
Linear algebra moves that picture into ten dimensions, where the intuition has to 
imagine the geometry (and gets it right) 


2. We move to matrix notation, writing the n unknowns as a vector x and the n equa- 
tions as Ax = b. We multiply A by “elimination matrices” to reach an upper trian- 
gular matrix U. Those steps factor A into L times U, where L is lower triangular. 
I will write down A and its factors for our example, and explain them at the right 


time: 
32 1 O;} J1 2 
Factorizati A= = = U. 6 
actorization 4 ; F j E | imes (6) 
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4x + 5y = 6 4x + 8y = 12 


One solution (zx, y) = (—1,2) Parallel: No solution Whole line of solutions 


Figure 1.1: The example has one solution. Singular cases have none or too many. 


First we have to introduce matrices and vectors and the rules for multiplication. 
Every matrix has a transpose A’. This matrix has an inverse A~!. 


3. In most cases elimination goes forward without difficulties. The matrix has an 
inverse and the system Ax = b has one solution. In exceptional cases the method 
will break down—either the equations were written in the wrong order, which is 
easily fixed by exchanging them, or the equations don’t have a unique solution. 


That singular case will appear if 8 replaces 5 in our example: 


Singular case kos 2y = 3 (7) 
Two parallel lines 4x + 8y = 6. 
Elimination still innocently subtracts 4 times the first equation from the second. But 


look at the result! 
(equation 2) — 4(equation 1) 0=-6. 


This singular case has no solution. Other singular cases have infinitely many solu- 
tions. (Change 6 to 12 in the example, and elimination will lead to 0 = 0. Now y 
can have any value,) When elimination breaks down, we want to find every possible 
solution. 


4. We need a rough count of the number of elimination steps required to solve a sys- 
tem of size n. The computing cost often determines the accuracy in the model. A 
hundred equations require a third of a million steps (multiplications and subtrac- 
tions). The computer can do those quickly, but not many trillions. And already 
after a million steps, roundoff error could be significant. (Some problems are sen- 
sitive; others are not.) Without trying for full detail, we want to see large systems 
that arise in practice, and how they are actually solved. 


The final result of this chapter will be an elimination algorithm that is about as effi- 
cient as possible. It is essentially the algorithm that is in constant use in a tremendous 
variety of applications. And at the same time, understanding it in terms of matrices—the 
coefficient matrix A, the matrices F for elimination and P for row exchanges, and the 
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final factors L and U—is an essential foundation for the theory. I hope you will enjoy 
this book and this course. 


1.2 The Geometry of Linear Equations 


The way to understand this subject is by example. We begin with two extremely humble 
equations, recognizing that you could solve them without a course in linear algebra. 
Nevertheless I hope you will give Gauss a chance: 


2h =" y 
x + y = 5. 


We can look at that system by rows or by columns. We want to see them both. 

The first approach concentrates on the separate equations (the rows). That is the 
most familiar, and in two dimensions we can do it quickly. The equation 2x — y = | is 
represented by a straight line in the x-y plane. The line goes through the points x = 1, 
y=landx= 5 y = 0 (and also through (2,3) and all intermediate points). The second 
equation x + y = 5 produces a second line (Figure 1.2a). Its slope is dy/dx = —1 and it 
crosses the first line at the solution. 

The point of intersection lies on both lines. It is the only solution to both equations. 
That point x = 2 and y = 3 will soon be found by “elimination.” 


y 2x —-y=1 
2 (column 1) 
Fas +3 (column 2) 


(a) Lines meet at x = 2, y=3 (b) Columns combine with 2 and 3 


Figure 1.2: Row picture (two lines) and column picture (combine columns). 


The second approach looks at the columns of the linear system. The two separate 
equations are really one vector equation: 


Column form x 
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The problem is to find the combination of the column vectors on the left side that 
produces the vector on the right side. Those vectors (2,1) and (—1,1) are represented 
by the bold lines in Figure 1.2b. The unknowns are the numbers x and y that multiply 
the column vectors. The whole idea can be seen in that figure, where 2 times column 
1 is added to 3 times column 2. Geometrically this produces a famous parallelogram. 
Algebraically it produces the correct vector (1,5), on the right side of our equations. 
The column picture confirms that x = 2 and y = 3. 

More time could be spent on that example, but I would rather move forward to n = 3. 
Three equations are still manageable, and they have much more variety: 


2 a. VPs cabs AY CS SS 
Three planes 4u — 6yv = 2 (1) 
—2u + Tv + 2w = 9. 


Again we can study the rows or the columns, and we start with the rows. Each equation 
describes a plane in three dimensions. The first plane is 2u-++-v+-w = 5, and it is sketched 
in Figure 1.3. It contains the points (3,0,0) and (0,5,0) and (0,0,5). It is determined 
by any three of its points—provided they do not lie on a line. 


Ww 


2u+u+w = 5 (sloqing plane) 


line of intersection: first two planes 


Figure 1.3: The row picture: three intersecting planes from three linear equations. 


Changing 5 to 10, the plane 2u-+-v+w = 10 would be parallel to this one. It contains 
(5,0,0) and (0,10,0) and (0,0,10), twice as far from the origin—which is the center 
point u = 0, v= 0, w = 0. Changing the right side moves the plane parallel to itself, and 
the plane 2u +v-+w =O goes through the origin. 
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The second plane is 4u —6v = —2. It is drawn vertically, because w can take any 
value. The coefficient of w is zero, but this remains a plane in 3-space. (The equation 
4u = 3, or even the extreme case u = 0, would still describe a plane.) The figure shows 
the intersection of the second plane with the first. That intersection is a line. In three 
dimensions a line requires two equations; in n dimensions it will require n — 1. 

Finally the third plane intersects this line in a point. The plane (not drawn) represents 
the third equation —2u+ 7v + 2w = 9, and it crosses the line at u= 1, v= 1, w= 2. That 
triple intersection point (1, 1,2) solves the linear system. 

How does this row picture extend into n dimensions? The n equations will con- 
tain n unknowns. The first equation still determines a “plane.” It is no longer a two- 
dimensional plane in 3-space; somehow it has “dimension” n— 1. It must be flat and 
extremely thin within n-dimensional space, although it would look solid to us. 

If time is the fourth dimension, then the plane ¢ = 0 cuts through four-dimensional 
space and produces the three-dimensional universe we live in (or rather, the universe as 
it was at f = 0). Another plane is z = 0, which is also three-dimensional; it is the ordinary 
x-y plane taken over all time. Those three-dimensional planes will intersect! They share 
the ordinary x-y plane at t = 0. We are down to two dimensions, and the next plane 
leaves a line. Finally a fourth plane leaves a single point. It is the intersection point of 4 
planes in 4 dimensions, and it solves the 4 underlying equations. 

I will be in trouble if that example from relativity goes any further. The point is that 
linear algebra can operate with any number of equations. The first equation produces an 
(n — 1)-dimensional plane in n dimensions, The second plane intersects it (we hope) in 
a smaller set of “dimension n — 2.” Assuming all goes well, every new plane (every new 
equation) reduces the dimension by one. At the end, when all n planes are accounted 
for, the intersection has dimension zero. It is a point, it lies on all the planes, and its 
coordinates satisfy all m equations. It is the solution! 


Column Vectors and Linear Combinations 


We turn to the columns. This time the vector equation (the same equation as (1)) is 


2 1 1 5 
Column form u}| 4 |+v}—6| +w]0} = |-2| =D. (2) 
—2 7 2 9 


Those are three-dimensional column vectors. The vector b is identified with the point 
whose coordinates are 5, —2, 9. Every point in three-dimensional space is matched to a 
vector, and vice versa. That was the idea of Descartes, who turned geometry into algebra 
by working with the coordinates of the point. We can write the vector in a column, or 
we can list its components as b = (5,—2,9), or we can represent it geometrically by an 
arrow from the origin. You can choose the arrow, or the point, or the three numbers. In 
six dimensions it is probably easiest to choose the six numbers. 
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We use parentheses and commas when the components are listed horizontally, and 
square brackets (with no commas) when a column vector is printed vertically. What 
really matters is addition of vectors and multiplication by a scalar (a number). In Figure 
1.4a you see a vector addition, component by component: 


5 0 0 5 
Vector addition O; + }-2}+ /0}; = |-2 
0) 0 9 9 


In the right-hand figure there is a multiplication by 2 (and if it had been —2 the vector 


[a] 2 [8] 


(column 3) 


columns 1 + 2 


(a) Add vectors along axes (b) Add columns 1 + 2 + (3 + 3) 


Figure 1.4: The column picture: linear combination of columns equals b. 


would have gone in the reverse direction): 


1 2 1 —2 
Multiplication by scalars ZO) SOs, S20 1) 
2 4 2 —4 


Also in the right-hand figure is one of the central ideas of linear algebra. It uses both 
of the basic operations; vectors are multiplied by numbers and then added. The result is 
called a linear combination, and this combination solves our equation: 


2 1 1 5 
Linear combination 1| 4 | +1)]-6} +2 /0] = |-2 
—2 7 2 9 


Equation (2) asked for multipliers u, v, w that produce the right side b. Those numbers 
are u= 1, v=1, w=2. They give the correct combination of the columns. They also 
gave the point (1, 1,2) in the row picture (where the three planes intersect). 
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Our true goal is to look beyond two or three dimensions into n dimensions. With n 
equations in n unknowns, there are n planes in the row picture. There are n vectors in 
the column picture, plus a vector b on the right side. The equations ask for a linear com- 
bination of the n columns that equals b. For certain equations that will be impossible. 
Paradoxically, the way to understand the good case is to study the bad one. Therefore 
we look at the geometry exactly when it breaks down, in the singular case. 


Row picture: Intersection of planes Column picture: Combination of columns 


The Singular Case 


Suppose we are again in three dimensions, and the three planes in the row picture do not 
intersect. What can go wrong? One possibility is that two planes may be parallel. The 
equations 2u-+v+w=5 and 4u+ 2v+ 2w = 11 are inconsistent—and parallel planes 
give no solution (Figure 1.5a shows an end view). In two dimensions, parallel lines 
are the only possibility for breakdown. But three planes in three dimensions can be in 
trouble without being parallel. 


KAA 


two parallel planes no intersection line of intersection all planes parallel 


(a) (b) (c) (d) 


Figure 1.5: Singular cases: no solution for (a), (b), or (d), an infinity of solutions for (c). 


The most common difficulty is shown in Figure 1.5b. From the end view the planes 
form a triangle. Every pair of planes intersects in a line, and those lines are parallel. The 
third plane is not parallel to the other planes, but it is parallel to their line of intersection. 
This corresponds to a singular system with b = (2,5,6): 


1 Sh ae a Ws Ss 
No solution, as in Figure 1.5b 2u + 3w 5 (3) 
3u + v + 4w = 6. 


The first two left sides add up to the third. On the right side that fails: 2+5 #6. Equation 
1 plus equation 2 minus equation 3 is the impossible statement 0 = 1. Thus the equations 
are inconsistent, as Gaussian elimination will systematically discover. 
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Another singular system, close to this one, has an infinity of solutions. When the 
6 in the last equation becomes 7, the three equations combine to give 0 = 0. Now the 
third equation is the sum of the first two. In that case the three planes have a whole line 
in common (Figure 1.5c). Changing the right sides will move the planes in Figure 1.5b 
parallel to themselves, and for b = (2,5,7) the figure is suddenly different. The lowest 
plane moved up to meet the others, and there is a line of solutions. Problem 1.5c is still 
singular, but now it suffers from too many solutions instead of too few. 

The extreme case is three parallel planes. For most right sides there is no solution 
(Figure 1.5d). For special right sides (like b = (0,0,0)!) there is a whole plane of 
solutions—because the three parallel planes move over to become the same. 

What happens to the column picture when the system is singular? it has to go wrong; 
the question is how, There are still three columns on the left side of the equations, and 
we try to combine them to produce b. Stay with equation (3): 


Singular case: Column picture 1 1 1 
Three columns in the same plane u{2)}+v]0| +w]3] =b. (4) 
Solvable only for b in that plane 3 1 4 


For b = (2,5,7) this was possible; for b = (2,5,6) it was not. The reason is that those 
three columns lie in a plane. Then every combination is also in the plane (which goes 
through the origin). If the vector D is not in that plane, no solution is possible (Figure 
1.6). That is by far the most likely event; a singular system generally has no solution. 
But there is a chance that b does lie in the plane of the columns. In that case there are too 
many solutions; the three columns can be combined in infinitely many ways to produce 
b. That column picture in Figure 1.6b corresponds to the row picture in Figure 1.5c. 


3 columns 
in a plane 


3 columns 


b not in place in a plane 


(a) no solution 


(b) infinity of solutions 


Figure 1.6: Singular cases: b outside or inside the plane with all three columns. 


How do we know that the three columns lie in the same plane? One answer is to finda 
combination of the columns that adds to zero. After some calculation, it is u = 3, v= 1, 
w = —2. Three times column | equals column 2 plus twice column 3. Column | is in 
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the plane of columns 2 and 3. Only two columns are independent. 

The vector b = (2,5,7) is in that plane of the columns—it is column 1 plus column 
3—so (1, 0, 1) is a solution. We can add an multiple of the combination (3,—1,—2) that 
gives b =(). So there is a whole line of solutions—as we know from the row picture. 

The truth is that we knew the columns would combine to give zero, because the rows 
did. That is a fact of mathematics, not of computation—and it remains true in dimension 
n. If the n planes have no point in common, or infinitely many points, then the n 
columns lie in the same plane. 

If the row picture breaks down, so does the column picture. That brings out the 
difference between Chapter | and Chapter 2. This chapter studies the most important 
problem—the nonsingular case—where there is one solution and it has to be found. 
Chapter 2 studies the general case, where there may be many solutions or none. In 
both cases we cannot continue without a decent notation (matrix notation) and a decent 
algorithm (elimination). After the exercises, we start with elimination. 


Problem Set 1.2 


1. For the equations x+y = 4, 2x — 2y = 4, draw the row picture (two intersecting 
lines) and the column picture (combination of two columns equal to the column 
vector (4,4) on the right side). 


2. Solve to find a combination of the columns that equals b: 


u-v-we= dD 
Triangular system v+we= bo 
w= b3. 


3. (Recommended) Describe the intersection of the three planes u+- v-+w+z= 6 and 
u+w+z=4 and u+w = 2 (all in four-dimensional space). Is it a line or a point or 
an empty set? What is the intersection if the fourth plane u = —1 is included? Find 
a fourth equation that leaves us with no solution. 


4. Sketch these three lines and decide if the equations are solvable: 


x 2S 2 
3 by 2 system x - y = 2 
ae © 


What happens if all right-hand sides are zero? Is there any nonzero choice of right- 
hand sides that allows the three lines to intersect at the same point? 


5. Find two points on the line of intersection of the three planes t = 0 and z = 0 and 
x+y+z-+t = 1 in four-dimensional space. 


10. 


11. 


12. 
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When b = (2,5,7), find a solution (u,v, w) to equation (4) different from the solution 
(1,0, 1) mentioned in the text. 


Give two more right-hand sides in addition to b = (2,5,7) for which equation (4) 
can be solved. Give two more right-hand sides in addition to b = (2,5,6) for which 
it cannot be solved. 


Explain why the system 


u+ v+ w = 2 
u+ 2v + 3w 1 
vy + 2w = 0 


is singular by finding a combination of the three equations that adds up to 0 = 1. 
What value should replace the last zero on the right side to allow the equations to 
have solutions—and what is one of the solutions? 


The column picture for the previous exercise (singular system) is 


1 1 1 
ujl} t+v]2| +w]3] =b. 
0 1 2) 


Show that the three columns on the left lie in the same plane by expressing the third 
column as a combination of the first two. What are all the solutions (u, v,w) if b is 
the zero vector (0,0,0)? 


(Recommended) Under what condition on yj, y2, y3 do the points (0,y1), (1,y2), 
(2,y3) lie on a straight line? 


These equations are certain to have the solution x = y = 0. For which values of a is 
there a whole line of solutions? 

Ge a. 2y =>) 

Do. a oye WO) 


Starting with x + 4y = 7, find the equation for the parallel line through x = 0, y = 0. 
Find the equation of another line that meets the first at x = 3, y= 1. 


Problems 13-15 are a review of the row and column pictures. 


13. 
14. 


Draw the two pictures in two planes for the equations x —2y = 0, x +y=6. 


For two linear equations in three unknowns x, y, z, the row picture will show (2 or 3) 
(lines or planes) in (two or three)-dimensional space. The column picture is in (two 
or three)-dimensional space. The solutions normally lie on a 
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15. 


16. 


17. 


18. 


19. 


20. 


21. 


22. 


23. 
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For four linear equations in two unknowns x and y, the row picture shows four 
. The column picture is in -dimensional space. The equations have no 
solution unless the vector on the right-hand side is a combination of 


Find a point with z = 2 on the intersection line of the planes x + y+ 3z = 6 and 
x—y-+z=4. Find the point with z = 0 and a third point halfway between. 


The first of these equations plus the second equals the third: 


x + y + Z 
RS ae 2 ae eZ 
25 Epes Sp ee Ds 


The first two planes meet along a line. The third plane contains that line, because 
if x, y, z satisfy the first two equations then they also . The equations have 
infinitely many solutions (the whole line L). Find three solutions. 


Move the third plane in Problem 17 to a parallel plane 2x + 3y + 2z = 9. Now the 
three equations have no solution—why not? The first two planes meet along the line 
L, but the third plane doesn’t that line. 


In Problem 17 the columns are (1,1,2) and (1,2,3) and (1,1,2). This is a “singular 
case” because the third column is . Find two combinations of the columns 
that give b = (2,3,5). This is only possible for b = (4,6,c) ifc = 


Normally 4 “planes” in four-dimensional space meet at a . Normally 4 col- 
umn vectors in four-dimensional space can combine to produce b. What combina- 
tion of (1,0,0,0), (1,1,0,0), (1,1,1,0), (1,1,1,1) produces b = (3,3,3,2)? What 4 
equations for x, y, z, tf are you solving? 


When equation | is added to equation 2, which of these are changed: the planes in 
the row picture, the column picture, the coefficient matrix, the solution? 


If (a,b) is a multiple of (c,d) with abcd 4 0, show that (a,c) is a multiple of (b,d). 
This is surprisingly important: call it a challenge question. You could use numbers 
first to see how a, b, c, and d are related. The question will lead to: 


IfA = e | has dependent rows then it has dependent columns. 


In these equations, the third column (multiplying w) is the same as the right side b. 
The column form of the equations immediately gives what solution for (u,v,w)? 


6u + Fv + 8w = 8 
4u + 5v + 9w 
2u — 2v + Iw = 7. 
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1.3. An Example of Gaussian Elimination 


The way to understand elimination is by example. We begin in three dimensions: 


2 Ae Ope ee aes Se 
Original system 4u- — 6v = —2 (1) 
—2u + Tv + 2w = 9. 


The problem is to find the unknown values of u, v, and w, and we shall apply Gaussian 
elimination. (Gauss is recognized as the greatest of all mathematicians, but certainly not 
because of this invention, which probably took him ten minutes. Ironically, it is the most 
frequently used of all the ideas that bear his name.) The method starts by subtracting 
multiples of the first equation from the other equations. The goal is to eliminate u from 
the last two equations. This requires that we 


(a) subtract 2 times the first equation from the second 


(b) subtract —1 times the first equation from the third. 


2 Ve a SS 
Equivalent system — 8 — 2w = -12 (2) 
8v + 3w = 14. 


The coefficient 2 is the first pivot. Elimination is constantly dividing the pivot into the 
numbers underneath it, to find out the right multipliers. 

The pivot for the second stage of elimination is —8. We now ignore the first equation. 
A multiple of the second equation will be subtracted from the remaining equations (in 
this case there is only the third one) so as to eliminate v. We add the second equation to 
the third or, in other words, we 


(c) subtract —1 times the second equation from the third. 


The elimination process is now complete, at least in the “forward” direction: 


2u+ v+w = 5 
Triangular system — 8v — 2w = -12 (3) 
lw = 2. 


This system is solved backward, bottom to top. The last equation gives w = 2. Sub- 
stituting into the second equation, we find v = 1. Then the first equation gives u = 1. 
This process is called back-substitution. 

To repeat: Forward elimination produced the pivots 2, —8, 1. It subtracted multiples 
of each row from the rows beneath, It reached the “triangular” system (3), which is 
solved in reverse order: Substitute each newly computed value into the equations that 
are waiting. 


14 Chapter 1 Matrices and Gaussian Elimination 


Remark. One good way to write down the forward elimination steps is to include the 
right-hand side as an extra column. There is no need to copy u and v and w and = at 
every step, so we are left with the bare minimum: 


Z. 1 Ae 3 2 ds , 4 ©) 2k A 5 
Ay 670 222) SS 0". Sb 2 2 0 8? 
2. ye 9 0 8 3 414 0 O 1 2 


At the end is the triangular system, ready for back-substitution. You may prefer this 
arrangement, which guarantees that operations on the left-hand side of the equations are 
also done on the right-hand side—because both sides are there together. 


In a larger problem, forward elimination takes most of the effort. We use multiples 
of the first equation to produce zeros below the first pivot. Then the second column is 
cleared out below the second pivot. The forward step is finished when the system is 
triangular; equation n contains only the last unknown multiplied by the last pivot. Back- 
substitution yields the complete solution in the opposite order—beginning with the last 
unknown, then solving for the next to last, and eventually for the first. 


By definition, pivots cannot be zero. We need to divide by them. 


The Breakdown of Elimination 


Under what circumstances could the process break down? Something must go wrong 
in the singular case, and something might go wrong in the nonsingular case. This may 
seem a little premature—after all, we have barely got the algorithm working. But the 
possibility of breakdown sheds light on the method itself. 


The answer is: With a full set of n pivots, there is only one solution. The system is 
non singular, and it is solved by forward elimination and back-substitution. But if'a zero 
appears in a pivot position, elimination has to stop—either temporarily or permanently. 
The system might or might not be singular. 


If the first coefficient is zero, in the upper left corner, the elimination of u from the 
other equations will be impossible. The same is true at every intermediate stage. Notice 
that a zero can appear in a pivot position, even if the original coefficient in that place 
was not zero. Roughly speaking, we do not know whether a zero will appear until we 
try, by actually going through the elimination process. 

In many cases this problem can be cured, and elimination can proceed. Such a system 
still counts as nonsingular; it is only the algorithm that needs repair. In other cases a 
breakdown is unavoidable. Those incurable systems are singular, they have no solution 
or else infinitely many, and a full set of pivots cannot be found. 
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Example 1. Nonsingular (cured by exchanging equations 2 and 3) 


u+tv+ Ww > __ u+tvpv+w > _ u+ v+ w 
2u + 2v + 5Bw = _-> BW) SS 2v + 4w = 
4u + 6v + 8w = 2v + 4w = 3w 


The system is now triangular, and back-substitution will solve it. 


Example 2. Singular (incurable) 


urVvy + Ww = u+tv+ Ww = 
2u + 2v + 5w = — 3w = 
4u + 4v + 8w = 4w 


There is no exchange of equations that can avoid zero in the second pivot position. The 
equations themselves may be solvable or unsolvable. If the last two equations are 3w = 6 
and 4w = 7, there is no solution. If those two equations happen to be consistent—as in 
3w = 6 and 4w = 8—then this singular case has an infinity of solutions. We know that 
w = 2, but the first equation cannot decide both u and v. 


Section 1.5 will discuss row exchanges when the system is not singular. Then the 
exchanges produce a full set of pivots. Chapter 2 admits the singular case, and limps 
forward with elimination. The 3w can still eliminate the 4w, and we will call 3 the 
second pivot. (There won’t be a third pivot.) For the present we trust all 1 pivot entries 
to be nonzero, without changing the order of the equations. That is the best case, with 
which we continue. 


The Cost of Elimination 


Our other question is very practical. How many separate arithmetical operations does 
elimination require, for n equations in n unknowns? If n is large, a computer is going to 
take our place in carrying out the elimination. Since all the steps are known, we should 
be able to predict the number of operations. 

For the moment, ignore the right-hand sides of the equations, and count only the 
operations on the left. These operations are of two kinds. We divide by the pivot to 
find out what multiple (say @) of the pivot equation is to be subtracted. When we do 
this subtraction, we continually meet a “multiply-subtract” combination; the terms in 
the pivot equation are multiplied by @, and then subtracted from another equation. 

Suppose we call each division, and each multiplication-subtraction, one operation. In 
column 1, it takes n operations for every zero we achieve—one to find the multiple @, 
and the other to find the new entries along the row. There are n — 1 rows underneath the 
first one, so the first stage of elimination needs n(n — 1) = n* —n operations. (Another 
approach to n* —n is this: All n? entries need to be changed, except the n in the first 
row.) Later stages are faster because the equations are shorter. 
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When the elimination is down to k equations, only k* — k operations are needed to 
clear out the column below the pivot—by the same reasoning that applied to the first 
stage, when k equaled n. Altogether, the total number of operations is the sum of k* — k 
over all values of k from | ton: 


Leftside = (1° + ---+n”)—(14+--- +n) = min Mens) _ mney) 


nm—n 


3 
Those are standard formulas for the sums of the first n numbers and the first n squares. 
Substituting n = 1 andn = 2 and n = 100 into the formula 3(n° —n), forward elimination 
can take no steps or two steps or about a third of a million steps: 


: , : aa Wee 
If n is at all large, a good estimate for the number of operations is xn”. 


If the size is doubled, and few of the coefficients are zero, the cost is multiplied by 8. 

Back-substitution is considerably faster. The last unknown is found in only one oper- 
ation (a division by the last pivot). The second to last unknown requires two operations, 
and so on. Then the total for back-substitution is 1 +2-+----++n. 

Forward elimination also acts on the right-hand side (subtracting the same multiples 
as on the left to maintain correct equations). This starts with n — 1 subtractions of the 
first equation. Altogether the right-hand side is responsible for n? operations—much 
less than the n>/3 on the left. The total for forward and back is 


Right side = [(n—1) +(n—2) +--+ J +[L4+24+--- +n] =n”. 


Thirty years ago, almost every mathematician would have guessed that a general sys- 
tem of order n could not be solved with much fewer than n> /3 multiplications. (There 
were even theorems to demonstrate it, but they did not allow for all possible methods.) 
Astonishingly, that guess has been proved wrong. There now exists a method that re- 
quires only Cn'°22’ multiplications! It depends on a simple fact: Two combinations of 
two vectors in two-dimensional space would seem to take 8 multiplications, but they can 
be done in 7. That lowered the exponent from log, 8, which is 3, to log, 7 ¥ 2.8. This 
discovery produced tremendous activity to find the smallest possible power of n. The 
exponent finally fell (at IBM) below 2.376. Fortunately for elimination, the constant C 
is so large and the coding is so awkward that the new method is largely (or entirely) of 
theoretical interest. The newest problem is the cost with many processors in parallel. 


Problem Set 1.3 


Problems 1-9 are about elimination on 2 by 2 systems. 
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. What multiple @ of equation 1 should be subtracted from equation 2? 


2h By = 
10x + 9y = 11. 


After this elimination step, write down the upper triangular system and circle the two 
pivots. The numbers | and 11 have no influence on those pivots. 


. Solve the triangular system of Problem | by back-substitution, y before x. Verify 
that x times (2,10) plus y times (3,9) equals (1,11). If the right-hand side changes 
to (4,44), what is the new solution? 


. What multiple of equation 2 should be subtracted from equation 3? 


2x — 4y = 6 
5. apa. Sy = 0; 


After this elimination step, solve the triangular system. If the right-hand side changes 
to (—6,0), what is the new solution? 


. What multiple @ of equation 1 should be subtracted from equation 2? 


ax + by = f 
cx + dy = g. 
The first pivot is a (assumed nonzero). Elimination produces what formula for the 


second pivot? What is y? The second pivot is missing when ad = bc. 


. Choose a right-hand side which gives no solution and another right-hand side which 
gives infinitely many solutions. What are two of those solutions? 


3x + 2y = 10 
6x + 4y 


. Choose a coefficient b that makes this system singular. Then choose a right-hand 
side g that makes it solvable. Find two solutions in that singular case. 


2x + by 16 
Ax: fe By =. @. 


. For which numbers a does elimination break down (a) permanently, and (b) tem- 
porarily? 

@yuse: Sy = 33 

4x + 6y = 6. 


Solve for x and y after fixing the second breakdown by a row exchange. 


10. 


11. 


12. 


13. 


14. 
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. For which three numbers k does elimination break down? Which is fixed by a row 


exchange? In each case, is the number of solutions 0 or 1 or co? 


kx + 3y = 6 
3h oe ky =" 0. 


. What test on b; and b2 decides whether these two equations allow a solution? How 


many solutions will they have? Draw the column picture. 


3x — 2y = by, 
6x — 4y = bo. 


Problems 10-19 study elimination on 3 by 3 systems (and possible failure). 
Reduce this system to upper triangular form by two row operations: 


20 ee yt Se eS. 38 
AR a dy aes De = 20 
= Dy + 2¢ = 0. 


Circle the pivots. Solve by back-substitution for z, y, x. 
Apply elimination (circle the pivots) and back-substitution to solve 
2x — 3y = 3 
Ay = Dye ee SS 
2x — y — 3z = 5. 
List the three row operations: Subtract times row from row : 


Which number d forces a row exchange, and what is the triangular system (not sin- 
gular) for that d? Which d makes this system singular (no third pivot)? 


20 246 Dye bo ag Se 0 
Ag oh dy aS goer 2 
y a4 = So: 


Which number 5 leads later to a row exchange? Which b leads to a missing pivot? 
In that singular case find a nonzero solution x, y, z. 


x + by =: 0 
x —- 2y —-z= 0 
y +z = 0. 


(a) Construct a 3 by 3 system that needs two row exchanges to reach a triangular 
form and a solution. 

(b) Construct a 3 by 3 system that needs a row exchange to keep going, but breaks 
down later. 
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15. If rows | and 2 are the same, how far can you get with elimination (allowing row 
exchange)? If columns | and 2 are the same, which pivot is missing? 


2x—y+z=0 2x +2y+z=0 
2x-—y+z=0 4x+4y+z=0 
4x+y+z=2 6x+6y+z=2. 


16. Construct a 3 by 3 example that has 9 different coefficients on the left-hand side, but 
rows 2 and 3 become zero in elimination. How many solutions to your system with 
b = (1,10, 100) and how many with b = (0,0,0)? 


17. Which number qg makes this system singular and which right-hand side ¢ gives it 
infinitely many solutions? Find the solution that has z= 1. 
x + 4y —- 22 = 
x + Ty -— 6 = 6 
3y + qz = t. 
18. (Recommended) It is impossible for a system of linear equations to have exactly two 
solutions. Explain why. 
(a) If (x,y,z) and (X,Y,Z) are two solutions, what is another one? 


(b) If 25 planes meet at two points, where else do they meet? 


19. Three planes can fail to have an intersection point, when no two planes are parallel. 
The system is singular if row 3 of A is a of the first two rows. Find a third 
equation that can’t be solved if x+y+z=0Oandx—2y—z=1. 


Problems 20-22 move up to 4 by 4 and 7 by n. 


20. Find the pivots and the solution for these four equations: 


2x 'y = 0 
RAS Zyeee ge = 0 
y +2z+ t = 0 

Z 2s: Qe =. 3 


21. If you extend Problem 20 following the 1, 2, 1 pattern or the —1, 2, —1 pattern, what 
is the fifth pivot? What is the nth pivot? 


22. Apply elimination and back-substitution to solve 


2u + 3v = 0 

11 a SV ee oe S'S 

2u —- v — 3w = 5. 
What are the pivots? List the three operations in which a multiple of one row is 
subtracted from another. 


20 


23. 


24. 


25. 


26. 


27. 


28. 
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For the system 
u+v+ w= 2 
u + 3v + 3w 0 
u + 3v + 5w = 2, 


what is the triangular system after forward elimination, and what is the solution? 
Solve the system and find the pivots when 


| a 
—u + 2v — w = 


noose 


— v + 2w—- Zz 
—- w +27 = 


You may carry the right-hand side as a fifth column (and omit writing u, v, w, z until 
the solution at the end). 


Apply elimination to the system 


u+yv+we= -2 
3u + 3v -—- w= 6 
uo—- v +w = -l. 


When a zero arises in the pivot position, exchange that equation for the one below it 
and proceed. What coefficient of v in the third equation, in place of the present —1, 
would make it impossible to proceed—and force elimination to break down? 


Solve by elimination the system of two equations 


x -—- y = O 
3x + 6y = 18. 


Draw a graph representing each equation as a straight line in the x-y plane; the lines 
intersect at the solution. Also, add one more line—the graph of the new second 
equation which arises after elimination. 


Find three values of a for which elimination breaks down, temporarily or perma- 
nently, in 

au+ u = 1 

4u + av = 2. 


Breakdown at the first step can be fixed by exchanging rows—but not breakdown at 
the last step. 


True or false: 


(a) If the third equation starts with a zero coefficient (it begins with Ou) then no 
multiple of equation 1 will be subtracted from equation 3. 


29. 


30. 


31. 


32. 
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(b) If the third equation has zero as its second coefficient (it contains Ov) then no 
multiple of equation 2 will be subtracted from equation 3. 


(c) If the third equation contains Ou and Ov, then no multiple of equation 1 or equa- 
tion 2 will be subtracted from equation 3. 


(Very optional) Normally the multiplication of two complex numbers 
(a+ib)(c +id) = (ac — bd) + i(bc +ad) 


involves the four separate multiplications ac, bd, be, ad. Ignoring i, can you compute 
ac — bd and bc + ad with only three multiplications? (You may do additions, such as 
forming a+ b before multiplying, without any penalty.) 


Use elimination to solve 


u+vp+w = 6 u+ y+ w= 7 
u + 2v + 2w = 11 and u + 2v+4+ 2w = 10 
2u + 3v — 4w = 3 2u + 3v — 4w = 3. 


For which three numbers a will elimination fail to give three pivots? 
ax+2y+3z=b, 
ax +ay+4z= b2 
ax + ay +az = b3. 

Find experimentally the average size (absolute value) of the first and second and third 


pivots for MATLAB’s lu(rand(3,3)). The average of the first pivot from abs(A(1, 1)) 
should be 0.5. 


1.4 Matrix Notation and Matrix Multiplication 


With our 3 by 3 example, we are able to write out all the equations in full. We can list 
the elimination steps, which subtract a multiple of one equation from another and reach 
a triangular matrix. For a large system, this way of keeping track of elimination would 
be hopeless; a much more concise record is needed. 


We now introduce matrix notation to describe the original system, and matrix mul- 


tiplication to describe the operations that make it simpler. Notice that three different 
types of quantities appear in our example: 


Nine coefficients 2u + v + w = 5 
Three unknowns 4u — 6v = —2 (1) 
Three right-hand sides —2u + Tv + 2w 


| 
Ney 
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On the right-hand side is the column vector b. On the left-hand side are the unknowns u, 
v, w. Also on the left-hand side are nine coefficients (one of which happens to be zero). 
It is natural to represent the three unknowns by a vector: 


1 
The unknown is x = | v The solution is x = | 1 
w 2 


The nine coefficients fall into three rows and three columns, producing a 3 by 3 matrix: 


2 1 1 
Coefficient matrix A=|4 -6 0 
2-7 2 


A is a square matrix, because the number of equations equals the number of unknowns. 
If there are n equations in n unknowns, we have a square n by n matrix. More generally, 
we might have m equations and n unknowns. Then A is rectangular, with m rows and n 
columns. It will be an “m by n matrix.” 

Matrices are added to each other, or multiplied by numerical constants, exactly as 
vectors are—one entry at a time. In fact we may regard vectors as special cases of 
matrices; they are matrices with only one column. As with vectors, two matrices can be 
added only if they have the same shape: 


1 1 ems, 2: «ll 4 2 
0; +)-3 1} =]0 1 2/3 0} = |6 0 
1 1 6 0 4 0 8 


Addition A + B 
Multiplication 2A 
Multiplication of a Matrix and a Vector 


We want to rewrite the three equations with three unknowns u, v, w in the simplified 
matrix form Ax = b. Written out in full, matrix times vector equals vector: 


2 1 1) |e 5 
Matrix form Ax = b 4 -6 O| |v] =]-2]. (2) 
—2 7 2) |w 9 


The right-hand side b is the column vector of “inhomogeneous terms.” The left-hand 
side is A times x. This multiplication will be defined exactly so as to reproduce the 
original system. The first component of Ax comes from “multiplying” the first row of A 
into the column vector x: 


u 
Row times column [2 1 1 i [2u+v+w| = 5 | : (3) 
w 


1.4 Matrix Notation and Matrix Multiplication 23 


The second component of the product Ax is 4u — 6v+ Ow, from the second row of A. The 
matrix equation Ax = b is equivalent to the three simultaneous equations in equation (1). 

Row times column is fundamental to all matrix multiplications. From two vectors it 
produces a single number. This number is called the inner product of the two vectors. 
In other words, the product of a 1 by n matrix (a row vector) and an n by | matrix (a 
column vector) is a 1 by 1 matrix: 


Inner product [2 1 1 


NO =—_- — 


= |2-141-1+41-2| = [5]. 


This confirms that the proposed solution x = (1, 1,2) does satisfy the first equation. 

There are two ways to multiply a matrix A and a vector x. One way is a row at a 
time, Each row of A combines with x to give a component of Ax. There are three inner 
products when A has three rows: 


6| |2 15945-54650 7 
1) 15) = |3-2+0-5+3-0| = |6|. (4) 
7 


1 1 
Ax by rows 3 0 
1 1 4] 10 1-2+1-5+4-0 


That is how Ax is usually explained, but the second way is equally important. In fact it is 
more important! It does the multiplication a column at a time. The product Ax is found 
all at once, as a combination of the three columns of A: 


1 1 6 q 
Ax by columns 2/3} +5/0/+0/3!] = |6 (5) 
1 1 4 7 


The answer is twice column 1 plus 5 times column 2. It corresponds to the “column 
picture” of linear equations. If the right-hand side b has components 7, 6, 7, then the 
solution has components 2, 5, 0. Of course the row picture agrees with that (and we 
eventually have to do the same multiplications). 

The column rule will be used over and over, and we repeat it for emphasis: 


1A Every product Ax can be found using whole columns as in equation (5). 
Therefore Ax is a combination of the columns of A. The coefficients are the 
components of x. 


To multiply A times x in n dimensions, we need a notation for the individual entries in 
A. The entry in the ith row and jth column is always denoted by a;;. The first subscript 
gives the row number, and the second subscript indicates the column. (In equation (4), 
az; 18 3 and aj3 is 6.) If A is an m by n matrix, then the index 7 goes from | to m—there 
are m rows—and the index j goes from 1 to n. Altogether the matrix has mn entries, and 
Amn is in the lower right corner. 
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One subscript is enough for a vector. The jth component of x is denoted by x;. (The 
multiplication above had x; = 2, x2 = 5, x3 = 0.) Normally x is written as a column 
vector—like an n by | matrix. But sometimes it is printed on a line, as in x = (2,5,0). 
The parentheses and commas emphasize that it is not a 1 by 3 matrix. It is a column 
vector, and it is just temporarily lying down. 

To describe the product Ax, we use the “sigma” symbol £ for summation: 

n 
Sigma notation The ith component of Ax is y Aj Xj. 
j=l 
This sum takes us along the ith row of A. The column index j takes each value from 1 
to n and we add up the results—the sum is aj}x1 + @j2X2 + +++ + inXn. 

We see again that the length of the rows (the number of columns in A) must match 
the length of x. An m by n matrix multiplies an n-dimensional vector (and produces 
an m-dimensional vector). Summations are simpler than writing everything out in full, 
but matrix notation is better. (Einstein used “tensor notation,” in which a repeated index 
automatically means summation. He wrote a;j;x; or even a! xj, without the X. Not being 
Einstein, we keep the ©.) 


The Matrix Form of One Elimination Step 


So far we have a convenient shorthand Ax = b for the original system of equations. 
What about the operations that are carried out during elimination? In our example, the 
first step subtracted 2 times the first equation from the second. On the right-hand side, 
2 times the first component of b was subtracted from the second component. The same 
result is achieved if we multiply b by this elementary matrix (or elimination matrix): 


1 0 0 
Elementary matrix E={|-2 1 0 
0 01 


This is verified just by obeying the rule for multiplying a matrix and a vector: 


1 0 0 5 a) 
Eb=|-—2 1 O| |-2] = |-12 
0 01 9 9 


The components 5 and 9 stay the same (because of the 1, 0, 0 and 0, 0, 1 in the rows of 
E). The new second component —12 appeared after the first elimination step. 

It is easy to describe the matrices like E, which carry out the separate elimination 
steps. We also notice the “identity matrix,” which does nothing at all. 


1B The identity matrix I, with 1s on the diagonal and Os everywhere else, 
leaves every vector unchanged. The elementary matrix E;; subtracts ¢ times 
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row j from row i. This £;; includes —¢ in row i, column j. 


1 0 0 1 0 0 bi 
IT=|0 1 O| hasIb=b F3,;=|0 1 O} has £3;b= bo 
00 1 —£ 0 1 b3 — tb, 


Ib = bis the matrix analogue of multiplying by 1. A typical elimination step 
multiplies by £3;. The important question is: What happens to A on the left- 
hand side? 


To maintain equality, we must apply the same operation to both sides of Ax = b. In 
other words, we must also multiply the vector Ax by the matrix E. Our original matrix 
E subtracts 2 times the first component from the second, After this step the new and 
simpler system (equivalent to the old) is just E(Ax) = Eb. It is simpler because of the 
zero that was created below the first pivot. It is equivalent because we can recover the 
original system (by adding 2 times the first equation back to the second). So the two 
systems have exactly the same solution x. 


Matrix Multiplication 


Now we come to the most important question: How do we multiply two matrices? There 
is a partial clue from Gaussian elimination: We know the original coefficient matrix A, 
we know the elimination matrix EF, and we know the result EA after the elimination step. 
We hope and expect that 


1 0 0 2. 1 * el 2. li sab 
E=}-—2 1 O| tmesA=]|4 —6 O} gives EA=|0 -8 -2 
0 0 1 —2 7 2 —2 7 2 


Twice the first row of A has been subtracted from the second row. Matrix multiplication 
is consistent with the row operations of elimination. We can write the result either as 
E(Ax) = Eb, applying E to both sides of our equation, or as (EA)x = Eb. The matrix 
EA is constructed exactly so that these equations agree, and we don’t need parentheses: 


Matrix multiplication (EA times x) equals (E times Ax). We just write FAx. 


This is the whole point of an “associative law” like 2 x (3 x 4) = (2 x 3) x 4. The law 
seems so obvious that it is hard to imagine it could be false. But the same could be said 
of the “commutative law” 2 x 3 = 3 x 2—and for matrices FA is not AE. 

There is another requirement on matrix multiplication. We know how to multiply Ax, 
a matrix and a vector. The new definition should be consistent with that one. When 
a matrix B contains only a single column x, the matrix-matrix product AB should be 
identical with the matrix-vector product Ax. More than that: When B contains several 
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columns b, b2, b3, the columns of AB should be Ab;, Ab2, Abs! 


by Ab, 
Multiplication by columns AB=A |b2| = |Abo 
b3 Ab3 


Our first requirement had to do with rows, and this one is concerned with columns. A 
third approach is to describe each individual entry in AB and hope for the best. In fact, 
there is only one possible rule, and I am not sure who discovered it. It makes everything 
work. It does not allow us to multiply every pair of matrices. If they are square, they 
must have the same size. If they are rectangular, they must not have the same shape; 
the number of columns in A has to equal the number of rows in B. Then A can be 
multiplied into each column of B. 

If A is m by n, and B is n by p, then multiplication is possible. The product AB will 
be m by p. We now find the entry in row i and column j of AB. 


1C The i, j entry of AB is the inner product of the ith row of A and the jth 
column of B. In Figure 1.7, the 3, 2 entry of AB comes from row 3 and column 
Des 

(AB)32 = a31b12 + a32b22 + 433b32 + a3aba2. (6) 


Row times 


column AB = 


Figure 1.7: A 3 by 4 matrix A times a 4 by 2 matrix B is a3 by 2 matrix AB. 


Note. We write AB when the matrices have nothing special to do with elimination. Our 
earlier example was EA, because of the elementary matrix EF. Later we have PA, or LU, 
or even LDU. The rule for matrix multiplication stays the same. 


2 SY WE 2 Ole | LP LO 
4 0] |5 -1 0] |4 8 Of 
The entry 17 is (2)(1) + (3)(5), the inner product of the first row of A and first column 
of B. The entry 8 is (4)(2) + (0)(—1), from the second row and second column. 

The third column is zero in B, so it is zero in AB. B consists of three columns side by 
side, and A multiplies each column separately. Every column of AB is a combination 


of the columns of A. Just as in a matrix-vector multiplication, the columns of A are 
multiplied by the entries in B. 


Example 1. 


AB= 
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Example 2. 


Row exchange matrix ae Ul oe a ne , 
1 O} |7 8 2. 3 


Example 3. The 1s in the identity matrix J leave every matrix unchanged: 
Identity matrix IA=A and BI=B. 


Important: The multiplication AB can also be done a row at a time. In Example 1, the 
first row of AB uses the numbers 2 and 3 from the first row of A. Those numbers give 
2[row 1] + 3[row 2] = [17 1 0]. Exactly as in elimination, where all this started, each 
row of AB is a combination of the rows of B. 

We summarize these three different ways to look at matrix multiplication. 


1D 
(i) Each entry of AB is the product of a row and a column: 
(AB) ;; = (row i of A) times (column j of B) 
(ii) Each column of AB is the product of a matrix and a column: 


column j of AB = A times (column j of B) 


(i11) Each row of AB is the product of a row and a matrix: 


row i of AB = (row i of A) times B. 


This leads hack to a key property of matrix multiplication. Suppose the shapes of 
three matrices A, B, C (possibly rectangular) permit them to be multiplied. The rows in 
A and B multiply the columns in B and C. Then the key property is this: 


1E Matrix multiplication is associative: (AB)C = A(BC). Just write ABC. 


AB times C equals A times BC. If C happens to be just a vector (a matrix with only one 
column) this is the requirement (EA)x = E(Ax) mentioned earlier. It is the whole basis 
for the laws of matrix multiplication. And if C has several columns, we have only to 
think of them placed side by side, and apply the same rule several times. Parentheses 
are not needed when we multiply several matrices. 

There are two more properties to mention—one property that matrix multiplication 
has, and another which it does not have. The property that it does possess is: 


1F Matrix operations are distributive: 


A(B+C)=AB+AC and (B+C)D=BD+CD. 
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Of course the shapes of these matrices must match properly—B and C have the same 
shape, so they can be added, and A and D are the right size for premultiplication and 
postmultiplication. The proof of this law is too boring for words. 

The property that fails to hold is a little more interesting: 


1G Matrix multiplication is not commutative: Usually FE 4 EF. 


Example 4. Suppose E subtracts twice the first equation from the second. Suppose F 
is the matrix for the next step, to add row | to row 3: 


1 00 10 0 
E=|-2 1 0 and F=]/0 1 0 
0 01 10 1 
These two matrices do commute and the product does both steps at once: 
1 00 
EF={|-2 1 0| =FE. 
1 01 


In either order, EF or FE, this changes rows 2 and 3 using row 1. 


Example 5. Suppose E is the same but G adds row 2 to row 3. Now the order makes a 
difference. When we apply E and then G, the second row is altered before it affects the 
third. If E comes after G, then the third equation feels no effect from the first. You will 
see a zero in the (3,1) entry of EG, where there is a —2 in GE: 


1 0 0 1 0 0 1 00 1 
GE=}|0 1 O| |-2 1 O} =|]-—2 1 O} but EG={-2 1 
011 0 O01 —2 141 0 11 


Thus EG 4 GE. A random example would show the same thing—most matrices don’t 
commute. Here the matrices have meaning. There was a reason for EF = FE, and a 
reason for EG ~ GE. It is worth taking one more step, to see what happens with all 
three elimination matrices at once: 


1 00 1 00 
GFE=|-2 1 0 and EFG=|-2 1 0 
-1 11 -1 11 


The product GFE is the true order of elimination. /t is the matrix that takes the original 
A to the upper triangular U. We will see it again in the next section. 

The other matrix EFG is nicer. In that order, the numbers —2 from EF and | from F 
and G were not disturbed. They went straight into the product. It is the wrong order for 
elimination. But fortunately it is the right order for reversing the elimination steps— 
which also comes in the next section. 

Notice that the product of lower triangular matrices is again lower triangular. 
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Problem Set 1.4 


1. Compute the products 


40 1] ]3 100 5 > oli 
O 1 O} [4] and |O 1 O| |—2] and ; ; ) 
40 1] {5 00 1 3 


For the third one, draw the column vectors (2,1) and (0,3). Multiplying by (1,1) 
just adds the vectors (do it graphically). 


2. Working a column at a time, compute the products 


aa 12°93) 10 4 3 
1 Hl and |4 5 6| {1} and {6 6 | 
1 


| 
6 7 8 9} 10 8 9 


Wl Nie 
| | 


3. Find two inner products and a matrix product: 


1 3 1 
1 =) 7] ~2] and E =) 7] 5| and |—2 3 5 i. 
7 1 7 


The first gives the length of the vector (squared). 


4. If an m by n matrix A multiplies an n-dimensional vector x, how many separate 
multiplications are involved? What if A multiplies an 1 by p matrix B? 


5. Multiply Ax to find a solution vector x to the system Ax = zero vector. Can you find 
more solutions to Ax = 0? 


a, OP “0 2 
Ax=|]0 2 -2] }1 
a 8 | 


6. Write down the 2 by 2 matrices A and B that have entries a;; = i+ j and bj; =(—1)'/. 
Multiply them to find AB and BA. 


7. Give 3 by 3 examples (not just the zero matrix) of 


(a) a diagonal matrix: a;; = Oif i F j. 
(b) a symmetric matrix: a;; = a;; for alli and /. 
(c) an upper triangular matrix: aj; = 0 if i> j. 


(d) a skew-symmetric matrix: a;; = —a;; for all i and /. 


8. Do these subroutines multiply Ax by rows or columns? Start with B(I) = 0: 


30 


10. 


11. 


12. 


13. 


14. 
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DO 101=1,N DO 10J=1,N 
DO 10J=1,N DO 101=1,N 
10 Bd) =B()+AQdJ) * X(J) 10 Bd) =B()+AGJ) * X(J) 


The outputs Bx = Ax are the same. The second code is slightly more efficient in 
FORTRAN and much more efficient on a vector machine (the first changes single 
entries B(/), the second can update whole vectors). 


. Ifthe entries of A are a;;, use subscript notation to write 


(a) the first pivot. 

(b) the multiplier 2;; of row 1 to be subtracted from row i. 

(c) the new entry that replaces a;; after that subtraction. 

(d) the second pivot. 

True or false? Give a specific counterexample when false. 

(a) If columns | and 3 of B are the same, so are columns 1 and 3 of AB. 
(b) If rows 1 and 3 of B are the same, so are rows | and 3 of AB. 

(c) If rows | and 3 of A are the same, so are rows | and 3 of AB. 

(d) (AB)? = AB’. 


The first row of AB is a linear combination of all the rows of B. What are the coeffi- 
cients in this combination, and what is the first row of AB, if 


A= 
QO -1 1 


2° Sp o4 ae 
| a B=10 1]? 
1 0 


The product of two lower triangular matrices is again lower triangular (all its entries 
above the main diagonal are zero). Confirm this with a 3 by 3 example, and then 
explain how it follows from the laws of matrix multiplication. 


By trial and error find examples of 2 by 2 matrices such that 
(a) A* = —J, A having only real entries. 

(b) B* = 0, although B F 0. 

(c) CD = —DC, not allowing the case CD = 0. 

(d) EF =0, although no entries of E or F are zero. 


Describe the rows of EA and the columns of AE if 


La 
0 1 


f= 


15. 


16. 


17. 


18. 


19. 


20. 


21. 
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Suppose A commutes with every 2 by 2 matrix (AB = BA), and in particular 


a b 
cd 


A= 


| commutes with B; = 


Show that a = d and b=c =0. If AB = BA for all matrices B, then A is a multiple 
of the identity. 


Let x be the column vector (1,0,...,0). Show that the rule (AB)x = A(Bx) forces the 
first column of AB to equal A times the first column of B. 


Which of the following matrices are guaranteed to equal (A + B)*? 


A’ +2AB+B*, A(A+B)+B(A+B), (A+B)(B+A), A?+AB+BA+B?. 


If A and B are n by n matrices with all entries equal to 1, find (AB);;. Summation 
notation turns the product AB, and the law (AB)C = A(BC), into 


(AB)ij=) andy [Zeun) c=) ax [Eoven) 
k iX\k k j 
Compute both sides if C is also n by n, with every cj = 2. 
A fourth way to multiply matrices is columns of A times rows of B: 
AB = (column 1)(row 1) +---+ (column n)(row n) = sum of simple matrices. 
Give a 2 by 2 example of this important rule for matrix multiplication. 
The matrix that rotates the x-y plane by an angle @ is 


HSE fe | . 


sin@ cos@ 
Verify that A( 0; )A(@2) = A(@; + 82) from the identities for cos(@; + 82) and sin(@; + 
62). What is A(@) times A(—80)? 


Find the powers A”, A> (A? times A), and B?, B°, C?, C?. What are A*, B‘, and C*? 


| 


1 
0 


> 

| 
NIP NIE 
NIP NIE 


and a-| | and C=AB= 


NIP NR 
NIP NIE 


Problems 22-31 are about elimination matrices. 


22. 


Write down the 3 by 3 matrices that produce these elimination steps: 


(a) E>; subtracts 5 times row | from row 2. 


(b) £32 subtracts —7 times row 2 from row 3. 


32 


23. 


24. 


25% 


26. 


27. 


28. 


29. 


30. 


31. 
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(c) P exchanges rows 1 and 2, then rows 2 and 3. 


In Problem 22, applying E; and then £3, to the column b = (1,0,0) gives E32E7)b = 
. Applying E32 before EF; gives E2,E32b = . When £32 comes first, row 
feels no effect from row , 


Which three matrices F;, £3;, £32 put A into triangular form U? 


1 1 
A=|4 6 and) £39F3;E2;A =U. 
2, 


or Oo 


Multiply those E’s to get one matrix M that does elimination: MA = U. 


Suppose a33 = 7 and the third pivot is 5. If you change a33 to 11, the third pivot is 


. If you change a33 to , there is zero in the pivot position. 


If every column of A is a multiple of (1,1, 1), then Ax is always a multiple of (1,1, 1). 
Do a3 by 3 example. How many pivots are produced by elimination? 


What matrix £3; subtracts 7 times row 1 from row 3? To reverse that step, R3; should 
7 times row to row . Multiply £3; by R31. 


(a) E>; subtracts row | from row 2 and then P)3 exchanges rows 2 and 3. What 
matrix M = P)3E>, does both steps at once? 

(b) Py3 exchanges rows 2 and 3 and then £3; subtracts row I from row 3. What 
matrix M = E3;P3 does both steps at once? Explain why the M’s are the same 
but the E’s are different. 

(a) What 3 by 3 matrix F}3 will add row 3 to row 1? 

(b) What matrix adds row 1 to row 3 and at the same time adds row 3 to row 1? 


(c) What matrix adds row | to row 3 and then adds row 3 to row 1? 


Multiply these matrices: 


0.905 2) [ks Ge 0 POO), PL 2-3 
O 1 O}| |4 5 6] |O 1 O} and kL Oy Wak: 3 
LOO) <8 (94 0D. —-1 0 1} J1 4 0 


2 =—1 0. 0 
he =f) 2.1) 0 
O =] 2 “el 
O° Ob -2 


Problems 32-44 are about creating and multiplying matrices 


32. 


33. 


34. 


35. 


36. 
37. 


38. 
39. 


40. 


41. 


42. 
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Write these ancient problems in a 2 by 2 matrix form Ax = b and solve them: 
(a) X is twice as old as Y and their ages add to 39, 
(b) (x,y) = (2,5) and (3,7) lie on the line y = mx-+-c. Find m and c. 


The parabola y = a+ bx+ cx’ goes through the points (x,y) = (1,4) and (2,8) and 
(3,14). Find and solve a matrix equation for the unknowns (a,b,c). 


Multiply these matrices in the orders EF and FE and E?: 
1 

E=\a 

b 


0) 
0 a 1 
1 Cc 


oOo F 
- Oo OS 


1 

0 

(a) Suppose all columns of B are the same. Then all columns of EB are the same, 
because each one is E times ; 


(b) Suppose all rows of B are [1 2 4]. Show by example that all rows of EB are not 
[1 2 4]. It is true that those rows are 


If E adds row | to row 2 and F adds row 2 to row 1, does EF equal FE? 


The first component of Ax is Ya jx; = a1X] + +++ + jnXn. Write formulas for the 
third component of Ax and the (1,1) entry of A’. 


If AB = / and BC = I, use the associative law to prove A = C. 


Ais 3 by 5, Bis 5 by 3, Cis 5 by 1, and D is 3 by 1. All entries are 1. Which of these 
matrix operations are allowed, and what are the results? 


BA AB ABD DBA A(B+C). 


What rows or columns or matrices do you multiply to find 

(a) the third column of AB? 

(b) the first row of AB? 

(c) the entry in row 3, column 4 of AB? 

(d) the entry in row 1, column | of CDE? 

(3 by 3 matrices) Choose the only B so that for every matrix A, 
(a) BA = 4A. 

(b) BA = 4B. 

(c) BA has rows | and 3 of A reversed and row 2 unchanged. 


(d) All rows of BA are the same as row | of A. 


True or false? 


34 


43. 


44. 
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(a) If A? is defined then A is necessarily square. 

(b) If AB and BA are defined then A and B are square. 
(c) If AB and BA are defined then AB and BA are square. 
(d) If AB = B then A = /. 


If A is m by n, how many separate multiplications are involved when 


(a) A multiplies a vector x with n components? 

(b) A multiplies an n by p matrix B? Then AB is m by p. 

(c) A multiplies itself to produce A*? Here m =n. 

To prove that (AB)C = A(BC), use the column vectors b,,...,b, of B. First suppose 
that C has only one column c with entries c1,..., Cy: 

AB has columns Ab,,...,Ab,, and Bc has one column c,b; +---+cyby. 

Then (AB)c = cjAbj + +--+ cpAbn, equals A(c1by +-+++Cybn) = A(Bc). 


Linearity gives equality of those two sums, and (AB)c = A(Bc). The same is true for 
all other of C. Therefore (AB)C = A(BC). 


Problems 45—49 use column-row multiplication and block multiplication. 


45. 


46. 


47. 


Multiply AB using columns times rows: 


1 0 1 
a= pal [3 o)=[2]B 3 ols = 
= 121 7 
a4 2 


Block multiplication separates matrices into blocks (submatrices). If their shapes 
make block multiplication possible, then it is allowed. Replace these x’s by numbers 
and confirm that block multiplication succeeds. 


x x|x x x|x 
4 B| = [AC + BD| and x x|x xX X|Xx 
ale; ee 


Draw the cuts in A and B and AB to show how each of the four multiplication rules 
is really a block multiplication to find AB: 

(a) Matrix A times columns of B. 

(b) Rows of A times matrix B. 

(c) Rows of A times columns of B. 


(d) Columns of A times rows of B. 


48. 


49. 


50. 


51. 


32. 


53. 


54. 


SD: 


56. 


Sis 
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Block multiplication says that elimination on column | produces 
a b 
0 |: 


Elimination for a 2 by 2 block matrix: When A~'A = I, multiply the first block row 
by CA~! and subtract from the second row, to find the “Schur complement” S: 


1 0 
—c/a I 


a 0b 
c D 


EA= 


I 0 
—CA! ] 


A B 
C.D 


A B 
O58 


With i? = —1, the product (A +iB)(x+ iy) is Ax+ iBx + iAy — By. Use blocks to 
separate the real part from the imaginary part that multiplies i: 


x 
Aa 


Ax — ‘1 real part 


? imaginary part 


Suppose you solve Ax = b for three special right-hand sides b: 


1 0 0 
Ax; = |0 and Axo= /1 and Ax3= |0 
0 0) 1 


If the solutions x;, x2, x3 are the columns of a matrix X, what is AX? 


If the three solutions in Question 51 are x; = (1,1,1) and x2 = (0,1,1) and x3 = 
(0,0, 1), solve Ax = b when b = (3,5,8). Challenge problem: What is A? 


1 1 1 1 
= A 
1 | b i 


If you multiply a northwest matrix A and a southeast matrix B, what type of matri- 
ces are AB and BA? “Northwest” and “southeast” mean zeros below and above the 
antidiagonal going from (1,7) to (n, 1). 


Find all matrices 


ye 


b 
‘ | that satisfy A 
c d 


Write 2x + 3y+z+5t = 8 as a matrix A (how many rows?) multiplying the column 
vector (x,y,z,t) to produce b. The solutions fill a plane in four-dimensional space. 
The plane is three-dimensional with no 4D volume. 


What 2 by 2 matrix P; projects the vector (x,y) onto the x axis to produce (x,0)? 
What matrix P) projects onto the y axis to produce (0,y)? If you multiply (5,7) by 
P, and then multiply by Po, you get ( ) and ( ). 


Write the inner product of (1,4,5) and (x,y,z) as a matrix multiplication Ax. A has 
one row. The solutions to Ax = 0 lie ona perpendicular to the vector . The 
columns of A are only in -dimensional space. 
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58. In MATLAB notation, write the commands that define the matrix A and the column 
vectors x and b. What command would test whether or not Ax = b? 


i 2 
3 4 
59. The MATLAB commands A = eye(3) and v = [8:5] produce the 3 by 3 identity 


matrix and the column vector (3,4,5). What are the outputs from A * v and v’ « v? 
(Computer not needed!) If you ask for v * A, what happens? 


A= 


60. If you multiply the 4 by 4 all-ones matrix A = ones(4,4) and the column v = 
ones(4,1), what is A * v? (Computer not needed.) If you multiply B = eye(4) 
+ ones(4,4) times w = zeros(4,1) + 2 * ones(4,1), what is B * w? 


61. Invent a3 by 3 magic matrix M with entries 1,2,...,9. All rows and columns and 
diagonals add to 15. The first row could be 8, 3, 4. What is M times (1, 1,1)? What 


is the row vector 1 1 1 times M? 


1.5 Triangular Factors and Row Exchanges 


We want to look again at elimination, to see what it means in terms of matrices. The 
starting point was the model system Ax = b: 


> oD) la 5 
Ax=|4 -6 0] |v| =]-2] =o. (1) 
ae ee Ma 9 


Then there were three elimination steps, with multipliers 2, —1, —1: 
Step 1. Subtract 2 times the first equation from the second; 

Step 2. Subtract —1 times the first equation from the third; 

Step 3. Subtract —1 times the second equation from the third. 


The result was an equivalent system Ux = c, with a new coefficient matrix U: 


2 1 1 u 5 
Upper triangular Ux=1]0 -—8 —2] |v| = |-12] =c. (2) 
0 0 1 w 2 


This matrix U is upper triangular—all entries below the diagonal are zero. 
The new right side c was derived from the original vector b by the same steps that 
took A into U. Forward elimination amounted to three row operations: 
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Start with A and b; 
Apply steps 1, 2, 3 in that order; 
End with U and c. 


Ux = cis solved by back-substitution. Here we concentrate on connecting A to U. 

The matrices EF for step 1, F for step 2, and G for step 3 were introduced in the 
previous section. They are called elementary matrices, and it is easy to see how they 
work. To subtract a multiple @ of equation 7 from equation i, put the number —£ into 
the (i, j) position. Otherwise keep the identity matrix, with 1s on the diagonal and Os 
elsewhere. Then matrix multiplication executes the row operation. 

The result of all three steps is GFEA = U. Note that E is the first to multiply A, 
then F,, then G. We could multiply GFE together to find the single matrix that takes A 
to U (and also takes b to c). It is lower triangular (zeros are omitted): 


1 1 1 1 
From A to U GFE = 1 1 =2 1 = |-2 1 . (3) 
1 1} {1 1 1 —-1 11 


This is good, but the most important question is exactly the opposite: How would we 
get from U back to A? How can we undo the steps of Gaussian elimination? 

To undo step | is not hard. Instead of subtracting, we add twice the first row to the 
second. (Not twice the second row to the first!) The result of doing both the subtraction 
and the addition is to bring back the identity matrix: 


Inverse of 100 1 00 100 
subtraction 2 1 0; |—2 1 O| =/0 1 O|. (4) 
is addition 00 1 0 O01 00 1 


One operation cancels the other. In matrix terms, one matrix is the inverse of the other. 
If the elementary matrix E has the number —¢ in the (i, j) position, then its inverse E~! 
has +4 in that position. Thus F —-lE =I, which is equation (4). 

We can invert each step of elimination, by using E~' and F~! and G~!. I think it’s 
not bad to see these inverses now, before the next section. The final problem is to undo 
the whole process at once, and see what matrix takes U back to A. 

Since step 3 was last in going from A to U, its matrix G must be the first to be 
inverted in the reverse direction. Inverses come in the opposite order! The second 
reverse step is F~! and the last is E~!: 


From U back to A E'F—G"u =Ais LU =A. (5) 


You can substitute GFEA for U, to see how the inverses knock out the original steps. 
Now we recognize the matrix L that takes U back to A. It is called L, because it is 
lower triangular. And it has a special property that can be seen only by multiplying the 
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three inverse matrices in the right order: 


1 1 1 1 
E"'F1gG1=]|2 1 1 1 =/|2 1 =L. (6) 
C} 4 1 es | -1 -11 


The special thing is that the entries below the diagonal are the multipliers ¢ = 2, —1, 
and —1. When matrices are multiplied, there is usually no direct way to read off the 
answer. Here the matrices come in just the right order so that their product can be 
written down immediately. If the computer stores each multiplier ¢;; the number that 
multiplies the pivot row j when it is subtracted from row i, and produces a zero in the i, 
j position—then these multipliers give a complete record of elimination. 

The numbers ¢;; fit right into the matrix L that takes U back to A. 


1H Triangular factorization A = LU with no exchanges of rows. L is lower 
triangular, with 1s on the diagonal. The multipliers @;; (taken from elimination) 
are below the diagonal. U is the upper triangular matrix which appears after 
forward elimination, The diagonal entries of U are the pivots. 


Example 1. 


ots 
3 


2 
oes to U = 
d S 


: with L= oy . Then LU =A. 
pe 3° 1 
Example 2. (which needs a row exchange) 


A= 


2 
3 i cannot be factored into A = LU. 


Example 3. (with all pivots and multipliers equal to 1) 


Et, 1 0 0; j1 1 1 
A= |1 2 2}=);1 1 0) |0 1 1] =LU 
Li 3 1 1 1) 4;0 0 1 


From A to U there are subtractions of rows. From U to A there are additions of rows. 


Example 4. (when U is the identity and L is the same as A) 


1 0O 0 
Lower triangular case A=!h, 1 O 
as ee a 


The elimination steps on this A are easy: (i) E subtracts @2; times row | from row 2, (ii) 
F subtracts £3; times row | from row 3, and (iii) G subtracts 239 times row 2 from row 3. 
The result is the identity matrix U = J. The inverses of F, F’, and G will bring back A: 
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E~! applied to F~! applied to G~! applied to J produces A. 


1 1 { 1 0 0 
fo, 1 times 1 times 1 equals |f.; 1 O 
1 631 1 £32 1 3, €32 1 


The order is right for the @’s to fall into position. This always happens! Note that 
parentheses in E~'F~'G~! were not necessary because of the associative law. 


A=LU: Then by n case 


The factorization A = LU is so important that we must say more. It used to be missing 
in linear algebra courses when they concentrated on the abstract side. Or maybe it was 
thought to be too hard—but you have got it. If the last Example 4 allows any U instead 
of the particular U = I, we can see how the rule works in general. The matrix L, applied 
to U, brings back A: 


1 O O} Jrow 1 of U 
A=LU fo, 1 O} |row 2 0f U| = original A. (7) 
l3; 37 1) |row30fU 


The proof is to apply the steps of elimination. On the right-hand side they take A to U. 
On the left-hand side they reduce L to J, as in Example 4. (The first step subtracts @ 
times (1,0,0) from the second row, which removes (2;.) Both sides of (7) end up equal 
to the same matrix U, and the steps to get there are all reversible. Therefore (7) is correct 
and A = LU. 

A = LU is so crucial, and so beautiful, that Problem 8 at the end of this section 
suggests a second approach. We are writing down 3 by 3 matrices, but you can see how 
the arguments apply to larger matrices. Here we give one more example, and then put 
A=LW to use. 


Example 5. (A = LU, with zeros in the empty spaces) 


1 -l 1 1 —-l 


That shows how a matrix A with three diagonals has factors L and U with two diagonals. 
This example comes from an important problem in differential equations (Section 1.7). 
The second difference in A is a backward difference L times a forward difference U. 
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One Linear System = Two Triangular Systems 


There is a serious practical point about A = LU. It is more than just a record of elimi- 
nation steps; L and U are the right matrices to solve Ax = b. In fact A could be thrown 
away! We go from b to c by forward elimination (this uses L) and we go from c to x by 
back-substitution (that uses U). We can and should do it without A: 


Splitting of Ax = b First Lco=b andthen Ux=c. (8) 


Multiply the second equation by L to give LUx = Lc, which is Ax = b. Each triangular 
system is quickly solved. That is exactly what a good elimination code will do: 


1. Factor (from A find its factors L and U). 


2. Solve (from L and U and b find the solution x). 
The separation into Factor and Solve means that a series of b’s can be processed. The 
Solve subroutine obeys equation (8): two triangular systems in n*/2 steps each. The 


solution for any new right-hand side b can be found in only n operations. That is 
far below the n> /3 steps needed to factor A on the left-hand side. 


Example 6. This is the previous matrix A with a right-hand side b = (1,1,1,1). 


xp = = ill 
= 2x. — = | 
Ax=b nae Uae oe splits into Lc = b and Ux =c. 
— x + 2x3 - x» = 1 
— x3 + 2x4 1 
Cl = i. ‘ 
— =. 1 2 
Lc=b oo is ee gives c= 
—- C2 + © an | 3 
—- 3 +c = 1 4 
ie SD 1 10 
x2. — x3 — a . 9 
Ux=c gives x= 
X307-— Xq4 = 3 
x4 4 4 


For these special “tridiagonal matrices,” the operation count drops from n? to 2n. You 
see how Lc = b is solved forward (c, comes before cz). This is precisely what happens 
during forward elimination. Then Ux = c is solved backward (x4 before x3). 


Remark 1. The LU form is “unsymmetric” on the diagonal: L has 1s where U has the 
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pivots. This is easy to correct. Divide out of U a diagonal pivot matrix D: 


dy 1 uy2/dy u13/d) 


d 
Factor out D U- 2 1 un3/dz 


(9) 
dn 1 

In the last example all pivots were d; = 1. In that case D = J. But that was very excep- 

tional, and normally LU is different from LDU (also written LDV). 


The triangular factorization can be written A = LDU, where Land U have \s on 
the diagonal and D is the diagonal matrix of pivots. 


Whenever you see LDU or LDV, it is understood that U or V has is on the diagonal— 
each row was divided by the pivot in D. Then L and U are treated evenly. An example 
of LU splitting into LDU is 
1 2 
—2 1 


sdb AL Al | 


That has the 1s on the diagonals of L and U, and the pivots 1 and —2 in D. 


A= = LDU. 


Remark 2. We may have given the impression in describing each elimination step, that 
the calculations must be done in that order. This is wrong. There is some freedom, and 
there is a “Crout algorithm” that arranges the calculations in a slightly different way. 
There is no freedom in the final L, D, and U. That is our main point: 


11 If A=L,D,U, and also A = LyD U2, where the L’s are lower triangular 
with unit diagonal, the U’s are upper triangular with unit diagonal, and the 
D’s are diagonal matrices with no zeros on the diagonal, then L; = Lz, D; = 
Dz, U; = U2. The LDU factorization and the LU factorization are uniquely 
determined by A. 


The proof is a good exercise with inverse matrices in the next section. 


Row Exchanges and Permutation Matrices 


We now have to face a problem that has so far been avoided: The number we expect to 
use as a pivot might be zero. This could occur in the middle of a calculation. It will 
happen at the very beginning if aj; = 0. A simple example is 


0 2) Iu b; 
Z in the pivot position = : 
ero in the pivot positi ‘ i | 


The difficulty is clear; no multiple of the first equation will remove the coefficient 3. 
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The remedy is equally clear. Exchange the two equations, moving the entry 3 up 
into the pivot. In this example the matrix would become upper triangular: 


3u+4v = bo 
2v = by 


To express this in matrix terms, we need the permutation matrix P that produces the 
row exchange. It comes from exchanging the rows of /: 


eee 


P has the same effect on b, exchanging b; and bz. The new system is PAx = Pb. The 
unknowns u and v are not reversed in a row exchange. 

A permutation matrix P has the same rows as the identity (in some order). There is 
a single “1” in every row and column. The most common permutation matrix is P = J (it 
exchanges nothing). The product of two permutation matrices is another permutation— 
the rows of J get reordered twice. 

After P = J, the simplest permutations exchange two rows. Other permutations ex- 
change more rows. There are 1! = (n)(n— 1)---(1) permutations of size n. Row | has 
n choices, then row 2 has n— 1 choices, and finally the last row has only one choice. We 
can display all 3 by 3 permutations (there are 3! = (3)(2)(1) = 6 matrices): 


1 1 1 
f= 1 Py, = |1 P32P 1 = 1 


Exchange rows 


Permutation P= and PA = 


P3) = 1 P39 = 1 Py, P32 = | 1 
1 1 1 
There will be 24 permutation matrices of order n = 4. There are only two permutation 


matrices of order 2, namely 
1 0 1 
and x ; 
0 1 1 0 


When we know about inverses and transposes (the next section defines A~! and A‘), 
we discover an important fact: P~! is always the same as P'. 


A zero in the pivot location raises two possibilities: The trouble may be easy to fix, 
or it may be serious. This is decided by looking below the zero. If there is a nonzero 
entry lower down in the same column, then a row exchange is carried out. The nonzero 
entry becomes the needed pivot, and elimination can get going again: 


0 ab d=0 == no first pivot 
A=10 0 ¢ a=0 ==> nosecond pivot 
a Za i c=0 == no third pivot. 
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If d = 0, the problem is incurable and this matrix is singular. There is no hope for a 
unique solution to Ax = b. If d is not zero, an exchange P;3 of rows | and 3 will move 
d into the pivot. However the next pivot position also contains a zero. The number a is 
now below it (the e above it is useless). If a is not zero then another row exchange P)3 is 
called for: 


and P»3P, 3A = 


cos;,g. 
ary 


e 
a 
0 
One more point: The permutation P)3P)3 will do both row exchanges at once: 


1 0 0} |O0 O 1 00 1 
P,3 acts first P33P}3= |0 0 1] |0 1 O} = {1 0 O| =P. 
0 1 O}; |1 O O 0 1 0 


If we had known, we could have multiplied A by P in the first place. With the rows in 
the right order PA, any nonsingular matrix is ready for elimination. 


Elimination in a Nutshell: PA = LU 


The main point is this: If elimination can be completed with the help of row exchanges, 
then we can imagine that those exchanges are done first (by P). The matrix PA will not 
need row exchanges. In other words, PA allows the standard factorization into L times 
U. The theory of Gaussian elimination can be summarized in a few lines: 


1J_ In the nonsingular case, there is a permutation matrix P that reorders 
the rows of A to avoid zeros in the pivot positions. Then Ax = b has a unique 
solution: 


With the rows reordered in advance, PA can be factored into LU. 


In the singular case, no P can produce a full set of pivots: elimination fails. 


In practice, we also consider a row exchange when the original pivot is near zero— 
even if it is not exactly zero. Choosing a larger pivot reduces the roundoff error. 

You have to be careful with L. Suppose elimination subtracts row 1 from row 2, 
creating (7; = 1. Then suppose it exchanges rows 2 and 3. If that exchange is done in 
advance, the multiplier will change to @3; = 1 in PA= LU. 


Example 7. 


1 
6; =U. (10) 
2 
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That row exchange recovers LU—but now @3; = | and 2; = 2: 


0 0 1 
0 1 and L=1|2 
1 0 1 


or Oo 


0 
0| and PA=LU. (11) 
1 


In MATLAB, A([r k] :) exchanges row k with row r below it (where the kth pivot has 
been found). We update the matrices L and P the same way. At the start, P = J and sign 
=+1: 


A([r k], :) = A(IK r], :); 
L({r k], 1:k-1) = L([k r], 1:k-1); 
P([r k], :) = P([k r], :); 
sign = -sign 
The “sign” of P tells whether the number of row exchanges is even (sign = +1) or odd 


(sign = —1). A row exchange reverses sign. The final value of sign is the determinant 
of P and it does not depend on the order of the row exchanges. 


To summarize: A good elimination code saves L and U and P. Those matrices carry 
the information that originally came in A—and they carry it in a more usable form. Ax = 
b reduces to two triangular systems. This is the practical equivalent of the calculation 
we do next—to find the inverse matrix A~! and the solution x = A~'b. 


Problem Set 1.5 


1. When is an upper triangular matrix nonsingular (a full set of pivots)? 


2. What multiple ¢32 of row 2 of A will elimination subtract from row 3 of A? Use the 
factored form 


What will be the pivots? Will a row exchange be required? 


3. Multiply the matrix L = E~'F~'G~! in equation (6) by GFE in equation (3): 


1 0 O 1 00 
2 1 O times —2 10 
Sl Sh] = OF: ab 


Multiply also in the opposite order. Why are the answers what they are? 
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. Apply elimination to produce the factors L and U for 


1 3 11 111 
A= g 1 and A=|/1 3 1 and A=|/1 4 4 
1 1 3 14 8 


. Factor A into LU, and write down the upper triangular system Ux = c which appears 
after elimination, for 


. Find E2 and E8 and E~! if 


1 

Ez ‘ 

i 

. Find the products FGH and HGF if (with upper triangular zeros omitted) 
i 1 1 
2 1 

Fe ee 0 1 _ 0 1 

00 1 02 1 00 1 
000 1 000 1 002 1 


. (Second proof of A = LU) The third row of U comes from the third row of A by 
subtracting multiples of rows 1 and 2 (of U!): 


row 3 of U = row 3 of A— £3 (row 1 of U) — ¢32(row 2 of U). 


(a) Why are rows of U subtracted off and not rows of A? Answer: Because by the 
time a pivot row is used, 


(b) The equation above is the same as 
row 3 of A = ¢3;(row | of U) + ¢32(row 2 of U) + 1 (row 3 of U). 
Which rule for matrix multiplication makes this row 3 of L times U? 
The other rows of LU agree similarly with the rows of A. 


. (a) Under what conditions is the following product nonsingular? 


1 O Of |d, 1 -l O 
A=|-1 1 0O d2 0 1 -tl 
0 -l1 1 d3||}0 O 1 


(b) Solve the system Ax = b starting with Lc = b: 
1 O Of} {ce 0 
—1 1 Of} Je2}] = }0} =5. 
0 -1 1] [c3 1 
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10. 


11. 


12. 


13. 


14. 


15. 


16. 


17. 


18. 
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(a) Why does it take approximately n7/2 multiplication-subtraction steps to solve 
each of Lc = b and Ux = c? 
(b) How many steps does elimination use in solving 10 systems with the same 60 by 
60 coefficient matrix A? 
Solve as two triangular systems, without multiplying LU to find A: 
10 O] |2 4 4] Ju 2 
LUx=]1 1 O} |O0 1 2] Jv} = |0 
101) /)0 0 1] |w 2 


How could you factor A into a product UL, upper triangular times lower triangular? 
Would they be the same factors as in A = LU? 


Solve by elimination, exchanging rows when necessary: 


uo + 4v + 2w = -—2 v+we= 0 
—2u — 8v + 3w = 32 and ust iy = 0 
vyv+w-= | u+vp+w =i il. 


Which permutation matrices are required? 


Write down all six of the 3 by 3 permutation matrices, including P = /. Identify their 
inverses, which are also permutation matrices. The inverses satisfy PP~' = J and are 
on the same list. 


Find the PA = LDU factorizations (and check them) for 


01 1 rE 2: I 
A=}]1 01 and A= |2 4 2 
23 4 Lf J 


Find a 4 by 4 permutation matrix that requires three row exchanges to reach the end 
of elimination (which is U = /). 


The less familiar form A = LPU exchanges rows only at the end: 


111 111 10 0//111 
A=1|11 3] ~L'A=|0 0 2| =PU=10 O 1| 10 3 6 
2.558 03 6 010/10 0 2 


What is L is this case? Comparing with PA = LU in Box 1J, the multipliers now stay 
in place (€2; is 1 and ¢3; is 2 when A = LPU). 
Decide whether the following systems are singular or nonsingular, and whether they 
have no solution, one solution, or infinitely many solutions: 

Vi We te 2 v—- w= 0 v+we= 1 
= 0 and u+y ae 
EN u =P ME EO. u +we=l, 


| 
< 
NS) 
pee) 
= 
am 
= 

| 
< 


19. 


20. 


21. 


22. 


23. 


24. 
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Which numbers a, b, c lead to row exchanges? Which make the matrix singular? 
[2 0 
ce 2 
6 4|- 


A= la 8 3 and A= 
Problems 20-31 compute the factorization A = LU (and also A = LDU). 


OPS 


Forward elimination changes [| }] x = b to a triangular [} |] x=: 


x+y =5 ee: rors fl SS 
ay —> : 
x + 2 = 7 yo ea.) | oe eee | 0 1 2 


That step subtracted 0); = times row | from row 2. The reverse step adds (21 
times row | to row 2. The matrix for that reverse step is L = . Multiply this Z 
times the triangular system |} |] x = [3] to get = . In letters, L multiplies 


Ux =c to give ‘ 


(Move to 3 by 3) Forward elimination changes Ax = b to a triangular Ux = c: 
x+y+z=5 xt+y+z=5 xty+z=5 


x+2y+3z=7 y+2z=2 y+2z=2 
x+3y+6z=11 2y+5z=6 ga 2: 
The equation z = 2 in Ux = c comes from the original x + 3y + 6z = 11 in Ax = b by 
subtracting 03; =__ times equation 1 and 037 = ____ times the final equation 2. 


Reverse that to recover {1 3 6 11] in [A b] from the final [1 1 1 5] and [0 1 2 2) 
and [0 0 1 2] in [U c]: 

Row 3 of [A b| = (£3; Row 1 + £32 Row 2+ 1 Row 3) of lu c| 
In matrix notation this is multiplication by L. SoA = LU and b = Lc. 


What are the 3 by 3 triangular systems Lc = b and Ux = c from Problem 21? Check 
that c = (5,2,2) solves the first one. Which x solves the second one? 


What two elimination matrices E; and £32 put A into upper triangular form £39F7,A = 
U? Multiply by ES and Es! to factor A into LU = Ey, Ee U ; 


What three elimination matrices E2;, £31;, E32 put A into upper triangular form 
F32F3,E2;A = U? Multiply by Ea Es and E.! to factor A into LU where L = 
jaa ceed aan Find L and U: 
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255 


26. 


27. 


28. 


29. 


30. 


Chapter 1 Matrices and Gaussian Elimination 


When zero appears in a pivot position, A = LU is not possible! (We need nonzero 
pivots d, f, iin U.) Show directly why these are both impossible: 


d 


1 1 0 1 e g 
0 1 1 Ol ld e 
palm leiffos] feat=fer if es 
12 1 mn il 


l 


Which number c leads to zero in the second pivot position? A row exchange is 
needed and A = LU is not possible. Which c produces zero in the third pivot position? 
Then a row exchange can’t help and elimination fails: 


1c 0 
Ass 20 Ae] 

3° 1 
What are Z and D for this matrix A? What is U in A = LU and what is the new U in 
A=LDU? 


A and B are symmetric across the diagonal (because 4 = 4). Find their triple factor- 
izations LDU and say how U is related to L for these symmetric matrices: 


1 4 0 
2 4 
A= and B=|4 12 4 
4 ]1 
0 4 0 


(Recommended) Compute L and U for the symmetric matrix 


Qa 90 &S 


Find four conditions on a, b, c, d to get A = LU with four pivots. 


Find L and U for the nonsymmetric matrix 


a QA & SN 
~ GS 


Q 


Find the four conditions on a, b, c, d, r, s, t to get A = LU with four pivots. 


31. 


32. 


33: 


34. 


35. 


36. 


37. 


38. 


39. 
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Tridiagonal matrices have zero entries except on the main diagonal and the two 
adjacent diagonals. Factor these into A = LU and A = LDV: 


1 1 0 a a 0 
Avs 1-2. al and A=|laa+b _ b 
012 0 bb b+e 


Solve the triangular system Lc = b to find c. Then solve Ux = c to find x: 


2 4 
and b= 5 ; 
0 1 11 


For safety find A = LU and solve Ax = b as usual. Circle c when you see it. 


C= 


1 0 
and U= 
a 


Solve Lc = b to find c. Then solve Ux = c to find x. What was A? 


100 111 4 
L=}1 10 and U=|01 1 and b= |5 
111 00 1 6 


If A and B have nonzeros in the positions marked by x, which zeros are still zero in 
their factors L and U? 


x xX xX Xx x x x 0 
Ae oD oy pele oe 
Ox x x x Ox x 
00x x Ox x x 


(Important) If A has pivots 2, 7, 6 with no row exchanges, what are the pivots for the 
upper left 2 by 2 submatrix B (without row 3 and column 3)? Explain why. 


Starting from a 3 by 3 matrix A with pivots 2, 7, 6, add a fourth row and column to 
produce M. What are the first three pivots for M, and why? What fourth row and 
column are sure to produce 9 as the fourth pivot? 


Use chol(pascal(5)) to find the triangular factors of MATLAB’s pascal(5). Row 
exchanges in [L, U] = lu(pascal(5)) spoil Pascal’s pattern! 


(Review) For which numbers c is A = LU impossible—with three pivots? 
b 2:0 


Aro o 16 
01 1 


Estimate the time difference for each new right-hand side b when n = 800. Create A 
= rand(800) and b = rand(800,1) and B = rand(800,9). Compare the times from 
tic; A\b; toc and tic; A\B; toc (which solves for 9 right sides). 


Problems 40-48 are about permutation matrices. 


50 


40. 


41. 


42. 


43. 


44. 


45. 


46. 


47. 


48. 
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There are 12 “even” permutations of (1,2,3,4), with an even number of exchanges. 
Two of them are (1,2,3,4) with no exchanges and (4,3,2,1) with two exchanges. 
List the other ten. Instead of writing each 4 by 4 matrix, use the numbers 4, 3, 2, 1 
to give the position of the | in each row. 


How many exchanges will permute (5,4,3,2,1) back to (1,2,3,4,5)? How many 
exchanges to change (6,5,4,3,2,1) to (1,2,3,4,5,6)? One is even and the other is 
odd. For (n,...,1) to (1,...,7), show that n = 100 and 101 are even, n = 102 and 
103 are odd. 


If P; and P) are permutation matrices, so is P,P). This still has the rows of J in some 
order. Give examples with P,P) ~ P,P; and P3P, = P,P3. 


(Try this question.) Which permutation makes PA upper triangular? Which permu- 
tations make P|AP) lower triangular? Multiplying A on the right by P) exchanges 
the of A. 


Find a3 by 3 permutation matrix with P? = J (but not P = J). Find a 4 by 4 permu- 
tation P with P* 4 J. 


If you take powers of a permutation, why is some P* eventually equal to /? Find a 5 
by 5 permutation P so that the smallest power to equal J is P®. (This is a challenge 
question. Combine a 2 by 2 block with a 3 by 3 block.) 


The matrix P that multiplies (x,y,z) to give (z,x,y) is also a rotation matrix. Find P 
and P>. The rotation axis a = (1,1, 1) doesn’t move, it equals Pa. What is the angle 
of rotation from v = (2,3, —5) to Pv = (—5,2,3)? 


If P is any permutation matrix, find a nonzero vector x so that (J — P)x = 0. (This 
will mean that J — P has no inverse, and has determinant zero.) 


If P has Is on the antidiagonal from (1,7) to (n,1), describe PAP. 


Inverses and Transposes 


The inverse of an n by n matrix is another n by n matrix. The inverse of A is written AW! 
(and pronounced “A inverse’’). The fundamental property is simple: [f you multiply by A 
and then multiply by A~!, you are back where you started: 


Inverse matrix If b=Ax then A 'b=x. 
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Thus A~!Ax =x. The matrix A~! times A is the identity matrix. Not all matrices have 
inverses. An inverse is impossible when Ax is zero and x is nonzero. Then A~! would 
have to get back from Ax = 0 to x. No matrix can multiply that zero vector Ax and 
produce a nonzero vector x. 

Our goals are to define the inverse matrix and compute it and use it, when Aq! 
exists—and then to understand which matrices don’t have inverses. 


1K The inverse of A is a matrix B such that BA = J and AB = J. There is at 
most one such B, and it is denoted by A7!: 


A'A=I and AA !=1. (1) 


Note 1. The inverse exists if and only if elimination produces n pivots (row exchanges 
allowed). Elimination solves Ax = b without explicitly finding A~!. 


Note 2. The matrix A cannot have two different inverses, Suppose BA = J and also 
AC = 1. Then B = C, according to this “proof by parentheses”: 

B(AC) = (BA)C gives BI=IC whichis B=C. (2) 
This shows that a left-inverse B (multiplying from the left) and a right-inverse C (multi- 
plying A from the right to give AC = J) must be the same matrix. 
Note 3. If A is invertible, the one and only solution to Ax = b is x = A7!b: 


Multiply Ax=b by A™!. Then x=A 'Ax=A™'b. 


Note 4. (Important) Suppose there is a nonzero vector x such that Ax =0. Then A 
cannot have an inverse. To repeat: No matrix can bring 0 back to x. 
If A is invertible, then Ax = 0 can only have the zero solution x = 0. 


Note 5. A 2 by 2 matrix is invertible if and only if ad — bc is not zero: 


=i 
a b 1 d —b 
2 by 2i SS 3 
Denner : ad — bc |“ a | ©) 


This number ad — bc is the determinant of A. A matrix is invertible if its determinant 
is not zero (Chapter 4). In MATLAB, the invertibility test is to find n nonzero pivots. 
Elimination produces those pivots before the determinant appears. 


Note 6. A diagonal matrix has an inverse provided no diagonal entries are zero: 
d\ 1/dj 
If A= me, then A! = es and AA! =I. 
di 1/dy 


When two matrices are involved, not much can be done about the inverse of A+ B. 
The sum might or might not be invertible. Instead, it is the inverse of their product 
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AB which is the key formula in matrix computations. Ordinary numbers are the same: 
(a+b)~! is hard to simplify, while 1/ab splits into 1/a times 1/b. But for matrices the 
order of multiplication must be correct—if ABx = y then Bx = A~'y and x = B-'A7!y. 
The inverses come in reverse order. 


1L A product AB of invertible matrices is inverted by B~!A7!: 


Inverse of AB (AB)! =B"'A"t. (4) 


Proof. To show that B~'A~! is the inverse of AB, we multiply them and use the associa- 
tive law to remove parentheses. Notice how B sits next to B7!: 


(AB)(B-'A~!) = ABB"'A7! =AIA"! =AA 1 =] 


(B-'A~!)(4B) = B-'A14B=B 'UB=B 'B=1. 


A similar rule holds with three or more matrices: 


Inverse of ABC. —- (ABC)! =C7'B'A7. 


We saw this change of order when the elimination matrices E, F, G were inverted to 
come back from U to A. In the forward direction, GFEA was U. In the backward 
direction, L = E~'F~'G~! was the product of the inverses. Since G came last, G7! 
comes first. Please check that A~! would be U~'GFE. 


The Calculation of 4~'!: The Gauss-Jordan Method 


Consider the equation AA~! = J. If it is taken a column at a time, that equation de- 
termines each column of A~!. The first column of A~! is multiplied by A, to yield the 
first column of the identity: Ax; = e;. Similarly Ax2 = e2 and Ax3 = e3 the e’s are the 
columns of J. In a3 by 3 example, A times A7! is /: 


ee | 1 0 0 
Ax; = ej 4 -6 0 x x9 8 = ler 2) es] = Oe lt) (5) 
=o 4 Ae 00 1 


Thus we have three systems of equations (or n systems). They all have the same coeffi- 
cient matrix A. The right-hand sides e1, e2, e3 are different, but elimination is possible 
on all systems simultaneously. This is the Gauss-Jordan method. Instead of stopping 
at U and switching to back-substitution, it continues by subtracting multiples of a row 
from the rows above. This produces zeros above the diagonal as well as below. When it 
reaches the identity matrix we have found A~!. 

The example keeps all three columns e1, e2, e3, and operates on rows of length six: 
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Example 1. Using the Gauss-Jordan Method to Find A7! 
2 11100 


[A ey ep e3}= 14 -60 01 0 
—2 72001 
2 1 1 1 O00 
pivot=2— |0 -8 -—2 -2 1 0 
0 8 3 1 O11 
2 1 1 1 00 
pivot —-8— |0 -8 -2 -2 1 0 =|u el 
0 O 1 -1141 


This completes the first half—forward elimination. The upper triangular U appears in 
the first three columns. The other three columns are the same as L~!. (This is the effect 
of applying the elementary operations GFE to the identity matrix.) Now the second half 
will go from U to J (multiplying by U~'). That takes L~! to U~'!L~! which is A7!. 
Creating zeros above the pivots, we reach A7!: 


as Re | a ame Ce 
Second half lu | sO: 28 to: ad. Be 
0 ee ae 
12 5 6 
- OF 73 78 
zeros above pivots— |0 —8 0 -4 3 2 
Oy Oe 3b Si. 4 
OOo a6 te 

divide by pivots |0 10 4 -3 —2 =|! Aq), 


001-1 1 1 


At the last step, we divided the rows by their pivots 2 and —8 and 1. The coefficient 
matrix in the left-hand half became the identity. Since A went to J, the same operations 
on the right-hand half must have carried J into A~!. Therefore we have computed the 
inverse. 

A note for the future: You can see the determinant — 16 appearing in the denominators 
of A~!. The determinant is the product of the pivots (2)(—8)(1). It enters at the end 
when the rows are divided by the pivots. 

Remark 1. In spite of this brilliant success in computing A~!, I don’t recommend it, I 
admit that A~! solves Ax = b in one step. Two triangular steps are better: 


x=A'b  separatesinto Lc=b and Ux=c. 


We could write c = L~'b and then x = U-'c = U~'L~'b. But note that we did not 
explicitly form, and in actual computation should not form, these matrices L~' and U~!. 
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It would be a waste of time, since we only need back-substitution for x (and forward 
substitution produced c). 

A similar remark applies to A~!; the multiplication A~'b would still take n? steps. It 
is the solution that we want, and not all the entries in the inverse. 


Remark 2. Purely out of curiosity, we might count the number of operations required 
to find A~!. The normal count for each new right-hand side is n*, half in the forward 
direction and half in back-substitution. With n right-hand sides e),...,e@, this makes n°. 
After including the n° /3 operations on A itself, the total seems to be 4n3/3. 


This result is a little too high because of the zeros in the e;. Forward elimination 
changes only the zeros below the 1. This part has only n — j components, so the count 
for e; is effectively changed to (n — j)”/2. Summing over all j, the total for forward 
elimination is n°/6. This is to be combined with the usual n>/3 operations that are 
applied to A, and the n(n? /2) back-substitution steps that finally produce the columns x; 


of A~!. The final count of multiplications for computing A~! is n?: 


. nr w n 3 

Operation count r oF 3 +n ) =n 

This count is remarkably low. Since matrix multiplication already takes n° steps, it 

requires as many operations to compute A” as it does to compute A~!! That fact seems 

almost unbelievable (and computing A? requires twice as many, as far as we can see). 
Nevertheless, if A~! is not needed, it should not be computed. 


Remark 3. In the Gauss-Jordan calculation we went all the way forward to U, before 
starting backward to produce zeros above the pivots. That is like Gaussian elimination, 
but other orders are possible. We could have used the second pivot when we were there 
earlier, to create a zero above it as well as below it. This is not smart. At that time 
the second row is virtually full, whereas near the end it has zeros from the upward row 
operations that have already taken place. 


Invertible = Nonsingular (n pivots) 


Ultimately we want to know which matrices are invertible and which are not. This 
question is so important that it has many answers. See the last page of the book! 

Each of the first five chapters will give a different (but equivalent) test for invertibility. 
Sometimes the tests extend to rectangular matrices and one-sided inverses: Chapter 2 
looks for independent rows and independent columns, Chapter 3 inverts AA! or ATA. 
The other chapters look for nonzero determinants or nonzero eigenvalues or nonzero 
pivots. This last test is the one we meet through Gaussian elimination. We want to show 
(in a few theoretical paragraphs) that the pivot test succeeds. 

Suppose A has a full set of n pivots. AA~! = J gives n separate systems Ax; = e; 
for the columns of A~!. They can be solved by elimination or by Gauss-Jordan. Row 
exchanges may be needed, but the columns of A~! are determined. 
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Strictly speaking, we have to show that the matrix A~! with those columns is also 
a left-inverse. Solving AA~' = / has at the same time solved A~'A = J, but why? A 
1-sided inverse of a square matrix is automatically a 2-sided inverse. To see why, 
notice that every Gauss-Jordan step is a multiplication on the left by an elementary 
matrix. We are allowing three types of elementary matrices: 


1. Ej; to subtract a multiple ¢ of row j from row i 
2. P;; to exchange rows i and j 
3. D (or D~') to divide all rows by their pivots. 
The Gauss-Jordan process is really a giant sequence of matrix multiplications: 
(D"!..-E-.-P---E)A=I. (6) 


That matrix in parentheses, to the left of A, is evidently a left-inverse! It exists, it equals 
the right-inverse by Note 2, so every nonsingular matrix is invertible. 

The converse is also true: If A is invertible, it has n pivots. In an extreme case that 
is clear: A cannot have a whole column of zeros. The inverse could never multiply a 
column of zeros to produce a column of J. In a less extreme case, suppose elimination 
starts on an invertible matrix A but breaks down at column 3: 


dj x x x 
Breakdown oO as A 
No pivot in column 3 10 00x 
0 0 0 x 


This matrix cannot have an inverse, no matter what the x’s are. One proof is to use 
column operations (for the first time?) to make the whole third column zero. By sub- 
tracting multiples of column 2 and then of column 1, we reach a matrix that is certainly 
not invertible. Therefore the original A was not invertible. Elimination gives a complete 
test: Ann by n matrix is invertible if and only if it has n pivots. 


The Transpose Matrix 


We need one more matrix, and fortunately it is much simpler than the inverse. The 
transpose of A is denoted by Al. Its columns are taken directly from the rows of A—the 
ith row of A becomes the ith column of A!: 


Transpose If A= 


214 0 
00 : then A'=]1 0 
4 3 


At the same time the columns of A become the rows of A/, If A is an m by n matrix, then 
A! is n by m. The final effect is to flip the matrix across its main diagonal, and the entry 
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in row i, column j of A! comes from row j, column i of A: 
Entries of AT —(A');; = Aji. (7) 


The transpose of a lower triangular matrix is upper triangular. The transpose of A! brings 
us back to A. 

If we add two matrices and then transpose, the result is the same as first transposing 
and then adding: (A+B)! is the same as A' + B'. But what is the transpose of a product 
AB or an inverse A~!? Those are the essential formulas of this section: 


1M 


(i) The transpose of AB is (AB)' = B'A!, 
(ii) The transpose of A~! is (A~!)? = (AT)—!. 


Notice how the formula for (AB)' resembles the one for (AB)~!. In both cases we 
reverse the order, giving B'A' and B~!A~!. The proof for the inverse was easy, but this 
one requires an unnatural patience with matrix multiplication. The first row of (AB)* is 
the first column of AB. So the columns of A are weighted by the first column of B. This 
amounts to the rows of A! weighted by the first row of B'. That is exactly the first row 
of B'A'. The other rows of (AB)! and B'A! also agree. 


ON 3S), 1333: 
1. Ail Oe BS SS 


3.2 11 ae 
Transpose to B'A’ = |3 2 | | cae 
a <5 


Start from AB = 


To establish the formula for (A~')", start from AA~! = J and A~'A = / and take trans- 
poses. On one side, J’ = J. On the other side, we know from part (i) the transpose of a 
product. You see how (A~!)! is the inverse of A', proving (ii): 


Inverse of A' = Transpose of A-' = (A!) TAT =I. (8) 


Symmetric Matrices 


With these rules established, we can introduce a special class of matrices, probably 
the most important class of all. A symmetric matrix is a matrix that equals its own 
transpose: A’ = A. The matrix is necessarily square. Each entry on one side of the 
diagonal equals its “mirror image” on the other side: a;; = a j;;. Two simple examples are 


A and D (and also A~!): 
2 1 0 _, 1] 8 2 
d A =-— ; 
0 ‘ on rl fe i 


Symmetric matrices A= 0 8 


| and D= 
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A symmetric matrix need not be invertible; it could even be a matrix of zeros. But if 
A”! exists it is also symmetric. From formula (ii) above, the transpose of A~! always 
equals (A')~!; for a symmetric matrix this is just A~'!. A~! equals its own transpose; it 
is symmetric whenever A is. Now we show that multiplying any matrix R by R" gives a 
symmetric matrix. 


Symmetric Products R'R, RR', and LDL" 


Choose any matrix R, probably rectangular. Multiply R! times R. Then the product R'R 
is automatically a square symmetric matrix: 


The transpose of R'R is R‘(R')', whichis R'R. (9) 


That is a quick proof of symmetry for R'R. Its i, j entry is the inner product of row i 
of R' (column i of R) with column j of R. The (j,7) entry is the same inner product, 
column j with column i. So R'R is symmetric. 

RR’ is also symmetric, but it is different from R'R. In my experience, most scientific 
problems that start with a rectangular matrix R end up with R'R or RR! or both. 
Example 2. R = [1 2] and R? = [3] produce R'R = [47] and RR’ = [5]. 

The product R'R is n by n. In the opposite order, RR! is m by m. Even if m = n, it is not 
very likely that R'R = RR". Equality can happen, but it’s not normal. 


Symmetric matrices appear in every subject whose laws are fair. “Each action has an 
equal and opposite reaction.” The entry a;; that gives the action of 7 onto j is matched 
by a;;. We will see this symmetry in the next section, for differential equations. Here, 
LU misses the symmetry but LDL" captures it perfectly. 


1N Suppose A = A! can be factored into A = LDU without row exchanges. 
Then U is the transpose of L. The symmetric factorization becomes A = LDL". 


The transpose of A = LDU gives A’ = U'D'L!. Since A = A‘, we now have two 
factorizations of A into lower triangular times diagonal times upper triangular. (L" is 
upper triangular with ones on the diagonal, exactly like U.) Since the factorization is 
unique (see Problem 17), L' must be identical to U. 


Fae 6 a 0 0) a (dma 
28] |2 1] \0 4|\o 1 


When elimination is applied to a symmetric matrix, A! = A is an advantage. The smaller 
matrices stay symmetric as elimination proceeds, and we can work with half the matrix! 
The lower right-hand corner remains symmetric: 


= LDL". 


abe a b ec 
bd 2) 2/0 -aa= eat 

c2 
Gre Ff 0 eb fe 
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The work of elimination is reduced from n° /3 to n>/6. There is no need to store entries 
from both sides of the diagonal, or to store both L and U. 


Problem Set 1.6 


1. Find the inverses (no special system required) of 
QZ dye 2 . sess cose —sin@ 
oe 50) 4 2 sin@ cos 


2. (a) Find the inverses of the permutation matrices 


A, = 


00 1 00 1 
P= 10 “LO and P=1]/1 0 0 
100 01 0 


(b) Explain for permutations why P™! is always the same as P'. Show that the 1s 
are in the right places to give PP! = /. 


3. From AB = C find a formula for A~!. Also find A~! from PA = LU. 


4. (a) IfA is invertible and AB = AC, prove quickly that B = C. 
(b) If A = [} }], find an example with AB = AC but B 4 C. 


5. If the inverse of A? is B, show that the inverse of A is AB. (Thus A is invertible 
whenever A? is invertible.) 


6. Use the Gauss-Jordan method to invert 


100 2 ek 0) 00 1 
Aj=]1 1 1], A2,=]-1 2 -l|, A3=/]0 1 1 
00 1 0 -1 2 111 
7. Find three 2 by 2 matrices, other than A = J and A = —/, that are their own inverses: 


AZ =I. 


8. Show that A = [} 3] has no inverse by solving Ax = 0, and by failing to solve 


1 lj {a b}  |1 0 
3 3} |e d| |0 1° 
9. Suppose elimination fails because there is no pivot in column 3: 


214 6 


03 8 5 
Missi ivot el 
issing pivo P 6088 


000 9 


10. 


11. 


12. 


13. 
14. 


15. 


16. 


17. 
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Show that A cannot be invertible. The third row of A~!, multiplying A, should give 
the third row [0 0 1 0] of A~!A =/. Why is this impossible? 


Find the inverses (in any legal way) of 


0001 1 0 0 0 baa 
0020 ae it 

fis . ie a2 Se He eae eS 
0300 0 -; 1 0 00a b 
4000 0 0 -21 00:0 a 


Give examples of A and B such that 


(a) A+B is not invertible although A and B are invertible. 
(b) A+B is invertible although A and B are not invertible. 
(c) all of A, B, and A+ B are invertible. 


(d) In the last case use A~!(A + B)B~! = B~'! + A7! to show that C= B-!+ A7! is 
also invertible—and find a formula for C~!. 


If A is invertible, which properties of A remain true for A~!? 
(a) A is triangular. (b) A is symmetric. (c) A is tridiagonal. (d) All entries are whole 
numbers. (e) All entries are fractions (including numbers like 3). 


If A = [}] and B = [4], compute A™B, B'A, AB’, and BA’. 


If B is square, show that A = B+ B' is always symmetric and K = B— B' is always 
skew-symmetric—which means that K' = —K. Find these matrices A and K when 
B = [13], and write B as the sum of a symmetric matrix and a skew-symmetric 
matrix. 


(a) How many entries can be chosen independently in a symmetric matrix of order 
n? 
(b) How many entries can be chosen independently in a skew-symmetric matrix 


(K! = —K) of order n? The diagonal of K is zero! 


(a) If A = LDU, with 1s on the diagonals of L and U, what is the corresponding 
factorization of A'? Note that A and A! (square matrices with no row exchanges) 
share the same pivots. 


(b) What triangular systems will give the solution to Aly = b? 


If A = L,D,U, and A = L2D2xUp, prove that L; = Ly, D, = Do, and U; = U>. If A 1s 
invertible, the factorization is unique. 


(a) Derive the equation Lo'LyDo = D, UU. | and explain why one side is lower 
q 1 p y 
triangular and the other side is upper triangular. 


(b) Compare the main diagonals and then compare the off-diagonals. 


60 


18. 


19. 


20. 


21. 


22. 


23. 


24. 


225 


26. 
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Under what conditions on their entries are A and B invertible? 


abe aboO 
A= |\d-e 0 B= lé -d-0 
f 0 0 0 0 e 
Compute the symmetric LDL! factorization of 
1 3 5 
a b 
A= 13 12 18 and A= b AE 
5 18 30 
Find the inverse of 
100 0 
1 
safe? 
Pas 
> 77 1 


(Remarkable) If A and B are square matrices, show that J — BA is invertible if /— AB 
is invertible. Start from B(J — AB) = (1 — BA)B. 


Find the inverses (directly or from the 2 by 2 formula) of A, B, C: 


Solve for the columns of A~! = ; i : 


bo sal =o] [ao 30 l= [if 


Show that [4 Al has no inverse by trying to solve for the column (x, y): 


pals [oa] mmf albl-bh 


(Important) If A has row 1 + row 2 = row 3, show that A is not invertible: 


(a) Explain why Ax = (1,0,0) cannot have a solution. 
(b) Which right-hand sides (b;,b2,b3) might allow a solution to Ax = b? 


(c) What happens to row 3 in elimination? 


If A has column | + column 2 = column 3, show that A is not invertible: 


27. 


28. 


29. 
30. 
31. 


32. 


33. 
34. 


35. 


36. 
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(a) Find a nonzero solution x to Ax = 0. The matrix is 3 by 3. 


(b) Elimination keeps column 1 + column 2 = column 3. Explain why there is no 
third pivot. 


Suppose A is invertible and you exchange its first two rows to reach B. Is the new 
matrix B invertible? How would you find B~! from A~!? 


If the product M = ABC of three square matrices is invertible, then A, B, C are 
invertible. Find a formula for B~! that involves M~! and A and C. 


Prove that a matrix with a column of zeros cannot have an inverse. 
Multiply [4 5] times [ 4. [2]. What is the inverse of each matrix if ad 4 bc? 


(a) What matrix FE has the same effect as these three steps? Subtract row 1 from row 
2, subtract row 1 from row 3, then subtract row 2 from row 3. 

(b) What single matrix L has the same effect as these three reverse steps? Add row 
2 to row 3, add row | to row 3, then add row 1 to row 2. 


Find the numbers a and b that give the inverse of 5 « eye(4) — ones(4,4): 


A a tly A ie 
-1 4 -1 -1] |babob 
=f =f <4: 1 b bab 
AL a ao bbba 


What are a and b in the inverse of 6 * eye(5) — ones(5,5)? 
Show that A = 4 « eye(4) — ones(4,4) is not invertible: Multiply A « ones(4,1). 


There are sixteen 2 by 2 matrices whose entries are 1s and 0s. How many of them 
are invertible? 


Problems 35-39 are about the Gauss-Jordan method for calculating A~!. 


Change J into A~! as you reduce A to J (by row operations): 


=f eos) me d= 5 6S) 


Follow the 3 by 3 text example but with plus signs in A. Eliminate above and below 
the pivots to reduce [A J] to [J A~!]: 
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37. 


38. 


39. 


40. 


41. 


42. 


43. 
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Use Gauss-Jordan elimination on [A J] to solve AA! =: 
Law D 100 
0 1 c| [er a2 #3|=|0 1 0 
001 00 1 
Invert these matrices A by the Gauss-Jordan method starting with [A /]: 


1 0 O ae Tene 
A=]2 1 3 and A=/1 2 2 
0 0 1 | ea 


True or false (with a counterexample if false and a reason if true): 


(a) A 4 by 4 matrix with a row of zeros is not invertible. 
(b) A matrix with Is down the main diagonal is invertible. 
(c) If A is invertible then A~! is invertible. 


(d) If A! is invertible then A is invertible. 
For which three numbers c is this matrix not invertible, and why not? 


2cc 
A= ".e° ee 
8 7 ¢ 


Prove that A is invertible if a 4 0 and a ¥ b (find the pivots and A~'): 


abb 
A=l!aa b 
aaa 


This matrix has a remarkable inverse. Find A~! by elimination on [A J]. Extend to a 
5 by 5 “alternating matrix” and guess its inverse: 


| ee 
v= 0 1 -1 1 
0 0 1 -!I 
0 0 0 1 


44. 


45. 


46. 


47. 


48. 


49. 


50. 


51. 


52; 


1.6 Inverses and Transposes 63 


If B has the columns of A in reverse order, solve (A — B)x = 0 to show that A — B is 
not invertible. An example will lead you to x. 


Find and check the inverses (assuming they exist) of these block matrices: 


I 0 
Ct 


A 0 
C D 


0 1 
I D 


Use inv(S) to invert MATLAB’s 4 by 4 symmetric matrix S = pascal(4). Create 
Pascal’s lower triangular A = abs(pascal(4,1)) and test inv(S) = inv(A’) « inv(A). 


If A = ones(4,4) and b = rand(4,1), how does MATLAB tell you that Ax = b has 
no solution? If b = ones(4,1), which solution to Ax = b is found by A\b? 


M~! shows the change in A! (useful to know) when a matrix is subtracted from A. 
Check part 3 by carefully multiplying MM~! to get I: 


1 M=I-w! and M-!=J+uy'/(1—v'u). 

2, M=A-w' and M-!=A!+A7'uytA7!/(1-vtA!u). 
3. M=I-UV and M-!=1,+U(In—VU)7'V. 

4. M=A—UW"'V and M7-!=A7!+A-!U(W-VA7!U)~!VAr!. 


The four identities come from the 1, 1 block when inverting these matrices: 


Ea 


Problems 49-55 are about the rules for transpose matrices. 


I, U A U 


V Wi 


Find AT and A~! and (A~!)? and (A‘)~! for 


loc 


A= : 
c 0 


1 
0 andalso A= 
9 3 


Verify that (AB) equals B'A™ but those are different from A'B!: 


p.|! 3 
0 1 


In case AB = BA (not generally true!), how do you prove that B'AT = ATB!? 


1 0 
Pama | 


i 3 
20° 


(a) The matrix ((AB)~)" comes from (A~!)' and (B~')". In what order? 


(b) If U is upper triangular then (U~!)" is triangular. 


Show that A* = 0 is possible but ATA = 0 is not possible (unless A = zero matrix). 
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535 


54. 


55. 


56. 


57. 


58. 


59. 


60. 


61. 
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(a) The row vector x! times A times the column y produces what number? 


tye fo ft 2 Jf] -— 
0 


(b) This is the row x'A = times the column y = (0,1,0). 
(c) This is the row x! = [0 1] times the column Ay = 


When you transpose a block matrix M = [4 8] the result is M? = . Test it. 
Under what conditions on A, B, C, D is the block matrix symmetric? 


Explain why the inner product of x and y equals the inner product of Px and Py. 
Then (Px)'(Py) =x" y says that P'P = J for any permutation. With x = (1,2,3) and 
y = (1,4,2), choose P to show that (Px)'y is not always equal to x'(P'y). 


Problems 56—60 are about symmetric matrices and their factorizations. 


If A =A! and B = B', which of these matrices are certainly symmetric? 
(a) A? — B? (b) (A+B)(A—B) (c) ABA (d) ABAB. 


If A =A! needs a row exchange, then it also needs a column exchange to stay sym- 
metric. In matrix language, PA loses the symmetry of A but recovers the sym- 
metry. 


(a) How many entries of A can be chosen independently, if A = A! is 5 by 5? 
(b) How do L and D (5 by 5) give the same number of choices in LDL"? 


Suppose R is rectangular (m by n) and A is symmetric (m by m). 


(a) Transpose R'AR to show its symmetry. What shape is this matrix? 


(b) Show why R'R has no negative numbers on its diagonal. 
Factor these symmetric matrices into A = LDL". The matrix D is diagonal: 


b 2 -1 O 
| and A=|/-1 2 -I1 
2 0. a1. 3 


1 
3 


1 
b 


A= 


; and A= 
2 


The next three problems are about applications of (Ax)'y = x'(A'y). 


Wires go between Boston, Chicago, and Seattle. Those cities are at voltages xp, xc, 
xs. With unit resistances between cities, the three currents are in y: 


YBC feed Oe) eB 
y= Ax is ycs | = 0 1 —1 XC 
YBS 1 O —I] |xs 
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63. 


64. 


65. 


66. 


67. 


68. 


69. 


70. 


71. 
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(a) Find the total currents A'y out of the three cities. 
(b) Verify that (Ax)'y agrees with x!(Aly)—-six terms in both. 


Producing x; trucks and x2 planes requires x; + 50x2 tons of steel, 40x; + 1000x2 
pounds of rubber, and 2x; + 50x2 months of labor. If the unit costs y1, y2, y3 are $700 
per ton, $3 per pound, and $3000 per month, what are the values of one truck and 
one plane? Those are the components of A'y. 


Ax gives the amounts of steel, rubber, and labor to produce x in Problem 62. Find A. 
Then (Ax)'y is the of inputs while x'(Ay) is the value of 


Here is a new factorization of A into triangular times symmetric: 
Start from A = LDU. Then A equals L(U*)~! times U'DU. 
Why is L(U')~! triangular? Its diagonal is all 1s. Why is U'DU symmetric? 


A group of matrices includes AB and A~! if it includes A and B. “Products and 
inverses stay in the group.” Which of these sets are groups? Lower triangularmatri- 
ces L with is on the diagonal, symmetric matrices S, positive matrices M, diagonal 
invertible matrices D, permutation matrices P. Invent two more matrix groups. 


If every row of a 4 by 4 matrix contains the numbers 0, 1, 2, 3 in some order, can the 
matrix be symmetric? Can it be invertible? 


Prove that no reordering of rows and reordering of columns can transpose a typical 
matrix. 


A square northwest matrix B is zero in the southeast corner, below the antidiagonal 
that connects (1,7) to (n,1). Will B' and B* be northwest matrices? Will B~! be 
northwest or southeast? What is the shape of BC = northwest times southeast? 
You are allowed to combine permutations with the usual L and U (southwest and 
northeast). 


Compare tic; inv(A); toc for A = rand(500) and A = rand(1000). The n° count 
says that computing time (measured by tic; toc) should multiply by 8 when n is 
doubled. Do you expect these random A to be invertible? 


I = eye(1000); A = rand(1000); B = triu(A); produces a random triangular matrix 
B. Compare the times for inv(B) and B\I. Backslash is engineered to use the zeros 
in B, while inv uses the zeros in J when reducing [B J] by Gauss-Jordan. (Compare 
also with inv(A) and A\J for the full matrix A.) 


Show that L~! has entries j/i for i < j (the —1, 2, —1 matrix has this L): 


1 OO OO 0O 1000 

1 1 

ade 4 
pelos Sy seid eee 

00 31 bias 
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Test this pattern for L = eye(5) — diag(1:5)\diag(1:4,—1) and inv(Z). 


1.7 Special Matrices and Applications 


This section has two goals. The first is to explain one way in which large linear systems 
Ax = bcan arise in practice. The truth is that a large and completely realistic problem in 
engineering or economics would lead us far afield. But there is one natural and important 
application that does not require a lot of preparation. 

The other goal is to illustrate, by this same application, the special properties that co- 
efficient matrices frequently have. Large matrices almost always have a clear pattern— 
frequently a pattern of symmetry, and very many zero entries. Since a sparse matrix 
contains far fewer than n7 pieces of information, the computations ought to be fast. We 
look at band matrices, to see how concentration near the diagonal speeds up elimination. 
In fact we look at one special tridiagonal matrix. 

The matrix itself can be seen in equation (6). It comes from changing a differential 
equation to a matrix equation. The continuous problem asks for u(x) at every x, and a 
computer cannot solve it exactly. It has to be approximated by a discrete problem—the 
more unknowns we keep, the better will be the accuracy and the greater the expense. 
As a simple but still very typical continuous problem, our choice falls on the differential 
equation 

du a 
os 

This is a linear equation for the unknown function u(x). Any combination C + Dx 
could be added to any solution, since the second derivative of C+ Dx contributes nothing. 
The uncertainty left by these two arbitrary constants C and D is removed by a “boundary 
condition” at each end of the interval: 


fo; Oxaet (1) 


u(O) =0, u(1) =0. (2) 


The result is a two-point boundary-value problem, describing not a transient but a steady- 
state phenomenon—the temperature distribution in a rod, for example, with ends fixed 
at 0°C and with a heat source f(x). 

Remember that our goal is to produce a discrete problem—in other words, a problem 
in linear algebra. For that reason we can only accept a finite amount of information about 
f(x), say its values at n equally spaced points x = h,x = 2h,...,x = nh. We compute 
approximate values u1,...,u, for the true solution u at these same points. At the ends 
x =O and x= 1 = (n+ 1)h, the boundary values are ug = 0 and uy,+1 = 0. 

The first question is: How do we replace the derivative d*u/dx*? The first derivative 
can be approximated by stopping Au/Ax at a finite stepsize, and not permitting / (or Ax) 
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to approach zero. The difference Au can be forward, backward, or centered: 


Au = wernt) a eta = EE ND 3) 


The last is symmetric about x and it is the most accurate. For the second derivative there 
is just one combination that uses only the values at x and x £h: 
du = A?u — u(x+h) —2u(x) +u(x—h) (4) 
dx? Aw? — h? 
This also has the merit of being symmetric about x. To repeat, the right-hand side ap- 
proaches the true value of d*u/dx as h — 0, but we have to stop at a positive h. 

At each meshpoint x = jh, the equation —d*u/dx* = f(x) is replaced by its discrete 
analogue (5). We multiplied through by h? to reach n equations Au = b: 


Second difference 


Difference equation — Uji t2uj—uj1=h’f(jh) forj=1,...,n. (5) 


The first and last equations (j = 1 and j = n) include ug = O and u,+4; = 0, which are 
known from the boundary conditions. These values would be shifted to the right-hand 
side of the equation if they were not zero. The structure of these n equations (5) can be 
better visualized in matrix form. We choose h = i to geta 5 by 5 matrix A: 


ae | uy f(h) 
—f 2.1 uy f (2h) 
Matrix equation a es u3} =h? | f(3h)|. 6) 
—1 2 =1] Iu f (4h) 
—1 2] [us f (5h) 


From now on, we will work with equation (6). It has a very regular coefficient matrix, 
whose order n can be very large. The matrix A possesses many special properties, and 
three of those properties are fundamental: 


1. The matrix A is tridiagonal. All nonzero entries lie on the main diagonal and the 
two adjacent diagonals. Outside this band all entries are a;; = 0. These zeros will 
bring a tremendous simplification to Gaussian elimination. 


2. The matrix is symmetric. Each entry a;; equals its mirror image a ;;, so that A'=A. 
The upper triangular U will be the transpose of the lower triangular L, and A = 
LDL’. This symmetry of A reflects the symmetry of d7u/dx?. An odd derivative 
like du/dx or d>u/dx* would destroy the symmetry. 


3. The matrix is positive definite. This extra property says that the pivots are positive. 
Row exchanges are unnecessary in theory and in practice. This is in contrast to the 
matrix B at the end of this section, which is not positive definite. Without a row 
exchange it is totally vulnerable to roundoff. 


Positive definiteness brings this whole course together (in Chapter 6)! 
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We return to the fact that A is tridiagonal. What effect does this have on elimination? 
The first stage of the elimination process produces zeros below the first pivot: 


2 -1 2 -1 
-1 2 -1 0 2% -1 
-1 2 -1 >] -1 2 -1 
-1 2 -1 -1 2 -1 

-1 2 -1 2 


Elimination 
on A: Step 1 


Compared with a general 5 by 5 matrix, that step displays two major simplifications: 
1. There was only one nonzero entry below the pivot. 
2. The pivot row was very short. 


The multiplier 2; = — came from one division. The new pivot 3 came from a single 
multiplication-subtraction. Furthermore, the tridiagonal pattern is preserved: Every 
stage of elimination admits the simplifications (a) and (b). 

The final result is the LDU = LDL’ factorization of A. Notice the pivots! 


2 1 
1 2 io 


— 
hin 
— 
| 
ns 


4 6 
= 6 1 


The L and U factors of a tridiagonal matrix are bidiagonal. The three factors together 
have the same band structure of three essential diagonals (3n — 2 parameters) as A. Note 
too that L and U are transposes of one another, as expected from the symmetry. The 
pivots 2/1, 3/2, 4/3, 5/4, 6/5 are all positive. Their product is the determinant of A: 
detA = 6. The pivots are obviously converging to 1, as n gets large. Such matrices make 
a computer very happy. 

These sparse factors L and U completely change the usual operation count. Elimina- 
tion on each column needs only two operations, as above, and there are n columns. In 
place of n° /3 operations we need only 2n. Tridiagonal systems Ax = b can be solved 
almost instantly. The cost of solving a tridiagonal system is proportional to n. 

A band matrix has aj; = 0 except in the band |i — j| < w (Figure 1.8). The “half 
bandwidth” is w = 1 for a diagonal matrix, w = 2 for a tridiagonal matrix, and w =n 
for a full matrix. For each column, elimination requires w(w — 1) operations: a row 
of length w acts on w — | rows below. Elimination on the n columns of a band matrix 
requires about wen operations. 

As w approaches n, the matrix becomes full, and the count is roughly n°. For an exact 
count, the lower right-hand corner has no room for bandwidth w. The precise number of 
divisions and multiplication-subtractions that produce L, D, and U (without assuming a 
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Figure 1.8: A band matrix A and its factors L and U. 


symmetric A) is P = +w(w —1)(3n—2w-+1). For a full matrix with w = n, we recover 
P= sn(n —1)(n+1). This is a whole number, since n — 1, n, and n+ 1 are consecutive 
integers, and one of them is divisible by 3. 

That is our last operation count, and we emphasize the main point. A finite-difference 
matrix like A has a full inverse. In solving Ax = b, we are actually much worse off 
knowing A~! than knowing L and U. Multiplying A~! by b takes n steps, whereas 4n 
are sufficient for the forward elimination and back-substitution that produce x = U~!c = 
U-'L'b=A™'b. 

We hope this example reinforced the reader’s understanding of elimination (which 
we now assume to be perfectly understood!). It is a genuine example of the large linear 
systems that are actually met in practice. The next chapter turns to the existence and the 
uniqueness of x, for m equations in n unknowns. 


Roundoff Error 


In theory the nonsingular case is completed. There is a full set of pivots (with row ex- 
changes). In practice, more row exchanges may be equally necessary—or the computed 
solution can easily become worthless. We will devote two pages (entirely optional in 
class) to making elimination more stable—why it is needed and how it is done. 

For a system of moderate size, say 100 by 100, elimination involves a third of a mil- 
lion operations (4n°). With each operation we must expect a roundoff error. Normally, 
we keep a fixed number of significant digits (say three, for an extremely weak computer). 
Then adding two numbers of different sizes gives an error: 


Roundoff Error 456 + .00123 — .457 loses the digits 2 and 3. 


How do all these individual errors contribute to the final error in Ax = b? 

This is not an easy problem. It was attacked by John von Neumann, who was the 
leading mathematician at the time when computers suddenly made a million operations 
possible. In fact the combination of Gauss and von Neumann gives the simple elimina- 
tion algorithm a remarkably distinguished history, although even von Neumann overes- 
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timated the final roundoff error. It was Wilkinson who found the right way to answer the 
question, and his books are now classics. 

Two simple examples will illustrate three important points about roundoff error. The 
examples are 


1. 1. 
1. 1.0001 


Ill-conditioned A= 1 1 


| Well-conditioned B= bite i . 


A is nearly singular whereas B is far from singular. If we slightly change the last entry 
of A to a22 = 1, it is singular. Consider two very close right-hand sides b: 

u+ -S 2 u+ ee 

and 

u + 1.000lv = 2 u + 1.0001lv = 2.0001 
The solution to the first is u = 2, v= 0. The solution to the second isu=v=1. A 
change in the fifth digit of b was amplified to a change in the first digit of the solution. No 
numerical method can avoid this sensitivity to small perturbations. The ill-conditioning 
can be shifted from one place to another, but it cannot be removed. The true solution is 
very sensitive, and the computed solution cannot be less so. 

The second point is as follows. 


10 Even a well-conditioned matrix like B can be ruined by a poor algorithm. 


We regret to say that for the matrix B, direct Gaussian elimination is a poor algorithm. 
Suppose .0001 is accepted as the first pivot. Then 10,000 times the first row is subtracted 
from the second. The lower right entry becomes —9999, but roundoff to three places 
would give —10,000. Every trace of the entry | would disappear: 


Elimination on B O0Olu+v = 1 a8 00lu+v 1 
with small pivot ut+tv = 2 —9999y = —9998. 


Roundoff will produce —10,000v = — 10,000, or v = 1. This is correct to three decimal 
places. Back-substitution with the right v = .9999 would leave u = 1: 


Correct result O00lu+.9999=1, or u=1. 
Instead, accepting v = 1, which is wrong only in the fourth place, we obtain u = 0: 
Wrong result O00lu+1l=1, or u=0. 


The computed u is completely mistaken. B is well-conditioned but elimination is vio- 
lently unstable. L, D, and U are completely out of scale with B: 


z_| 1 9} |.0001 0 1 10,000 
~ {10,000 1} 0 9999] }o 1. |’ 


The small pivot .0001 brought instability, and the remedy is clear—exchange rows. 
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1P A small pivot forces a practical change in elimination. Normally we 
compare each pivot with all possible pivots in the same column. Exchanging 
rows to obtain the largest possible pivot is called partial pivoting. 


For B, the pivot .0001 would be compared with the possible pivot I below it. A row 
exchange would take place immediately. In matrix terms, this is multiplication by a 
permutation matrix P = [9 }]. The new matrix C = PB has good factors: 


Ge LDP > Ohi 1 1 
~ |.0001 1] |.0001 1] /0 .9999] Jo 1 


The pivots for C are 1 and .9999, much better than .0001 and —9999 for B. 

The strategy of complete pivoting looks also in all later columns for the largest pos- 
sible pivot. Not only a row but also a column exchange may be needed. (This is 
postmultiplication by a permutation matrix.) The difficulty with being so conservative 
is the expense, and partial pivoting is quite adequate. 

We have finally arrived at the fundamental algorithm of numerical linear algebra: 
elimination with partial pivoting. Some further refinements, such as watching to see 
whether a whole row or column needs to be resealed, are still possible. But essentially 
the reader now knows what a computer does with a system of linear equations. Com- 
pared with the “theoretical” description—find A~!, and multiply A~'b—our description 
has consumed a lot of the reader’s time (and patience). I wish there were an easier way 
to explain how x is actually found, but I do not think there is. 


Problem Set 1.7 


1. Write out the LDU = LDL’ factors of A in equation (6) when n = 4. Find the deter- 
minant as the product of the pivots in D. 


2. Modify a;; in equation (6) from a,; = 2 to aj; = 1, and find the LDU factors of this 
new tridiagonal matrix. 


3. Find the 5 by 5 matrix Ag (4 = é) that approximates 


du du du 
oe ay 0) = Mt) = 9 
replacing these boundary conditions by ug = u; and ug = us. Check that your Ag 
times the constant vector (C,C,C,C,C), yields zero; Ao is singular. Analogously, if 
u(x) is a solution of the continuous problem, then so is u(x) +C. 


4. Write down the 3 by 3 finite-difference matrix equation (h = i) for 
du 


aa teas u(0) = u(1) =0. 
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. Withh= \ and f(x) = 427 sin27x, the difference equation (5) is 


2: sh? 0) Tae sic 
T 

—1 2 —l i) = Ae 0 

0 —] 2 U3 —1 


Solve for uv}, u2, v3 and find their error in comparison with the true solution u = 


. ie een | a8 
sin27x atx = 4,x= 5, and x = a 


. What 5 by 5 system replaces (6) if the boundary conditions are changed to u(0) = 1, 


u(1) =0? 
Problems 7-11 are about roundoff error and row exchanges. 


. Compute H~! in two ways for the 3 by 3 Hilbert matrix 


H= 


BIR WIRE NIE 
MR BIR DR 


WIR N|— — 


first by exact computation and second by rounding off each number to three figures. 
This matrix H is ill-conditioned and row exchanges don’t help. 


. For the same matrix H, compare the right-hand sides of Hx = b when the solutions 


are x = (1,1,1) and x = (0,6, —3.6). 


. Solve Hx = b= (1,0,...,0) for the 10 by 10 Hilbert matrix with h;; = 1/(i+ j—1), 


using any computer code for linear equations. Then change an entry of b by .0001 
and compare the solutions. 


Compare the pivots in direct elimination to those with partial pivoting for 


A= 


001 0 
1 1000] 


(This is actually an example that needs rescaling before elimination.) 


Explain why partial pivoting produces multipliers ¢;; in L that satisfy |¢;;| < 1. Can 
you construct a 3 by 3 example with all |a;;| < 1 whose last pivot is 4? This is the 
worst possible, since each entry is at most doubled when |é;;| < 1. 


Review Exercises 


1.1 (a) Write down the 3 by 3 matrices with entries 


ajj=i-J and bij = =. 


1.2 


1.3 


1.4 


1.5 
1.6 


1.7 


1.8 


1.9 
1.10 


1.11 


1.12 
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(b) Compute the products AB and BA and A?. 


For the matrices 


compute AB and BA and A~! and B~! and (AB)~!. 


Find examples of 2 by 2 matrices with a,2 = 5 for which (a) A? = I. (b) 
A l=Al!. (c) A? =A. 


Solve by elimination and back-substitution: 


u + w= 4 v+we-= 0 
u+ iy ae | and u +w-= 0 
u+vp+w= 6 u+ioyv = 6: 


Factor the preceding matrices into A = LU or PA= LU. 


(a) There are sixteen 2 by 2 matrices whose entries are 1s and 0s. How many are 
invertible? 

(b) (Much harder!) If you put 1s and Os at random into the entries of a 10 by 10 
matrix, is it more likely to be invertible or singular? 


There are sixteen 2 by 2 matrices whose entries are 1s and —ls. How many are 
invertible? 


How are the rows of FA related to the rows of A in the following cases? 


1 0 0 ie 4 00 1 
E= 1/0 2 0 or E= 0°00 or E=1|0 1 0 
40 1 1 0 0 
Write down a 2 by 2 system with infinitely many solutions. 
Find inverses if they exist, by inspection or by Gauss-Jordan: 
10 1 2: Le 1 1 -2 
A=]|1 1 0} and A=]J1 2 1 and A=/}1 -—2 1 
01 1 0 12 —2 1 1 


If E is 2 by 2 and it adds the first equation to the second, what are E* and E® and 
8E? 


True or false, with reason if true or counterexample if false: 


(1) If A is invertible and its rows are in reverse order in B, then B is invertible. 
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(2) If A and B are symmetric then AB is symmetric. 

(3) If A and B are invertible then BA is invertible. 

(4) Every nonsingular matrix can be factored into the product A = LU of a lower 
triangular Z and an upper triangular U. 


1.13 Solve Ax = b by solving the triangular systems Lc = b and Ux =: 


1 0 O| |2 2 4 0 
A=lLU= 1/4 1 0] |0 1 3), b= |0 
101; /0 01 1 
What part of A~! have you found, with this particular b? 
1.14 If possible, find 3 by 3 matrices B such that 
(1) BA = 2A for every A. 
(2) BA = 2B for every A. 
(3) BA has the first and last rows of A reversed. 
(4) BA has the first and last columns of A reversed. 
1.15 Find the value for c in the following n by n inverse: 
n -1l - -1 1 
—] - =] 1 - | 
if A= ‘ then A} = . 
. : | n+1 - ] 
—1 -1 -l n ee dh TB, 
1.16 For which values of k does 
ke oe Sar 
ee ky SS 
have no solution, one solution, or infinitely many solutions? 
1.17 Find the symmetric factorization A = LDL! of 
1 2.0 
a b 
A=|2 6 4 wat =| Oh 
0411 i 
1.18 Suppose A is the 4 by 4 identity matrix except for a vector v in column 2: 


1 Vi 0 O 

0 w 0 : 
O ago Ol’ 
0 v4 O ‘| 


1.19 


1.20 


1.21 
1.22 


1.23 


1.24 


1.25 


1.26 
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(a) Factor A into LU, assuming v2 4 0. 
(b) Find A~!, which has the same form as A. 


Solve by elimination, or show that there is no solution: 


u+ v+ w= 0 u+ v+ we= 0 
u+ 2v4+ 3w = 0 and u+ ut 3w= 0 
3u + 5v + Iw= il 3u + 5v + Tw = 1. 


The n by n permutation matrices are an important example of a “group.” If you 
multiply them you stay inside the group; they have inverses in the group; the identity 
is in the group; and the law P; (P)P3) = (P;P2)P3 is true—because it is true for all 
matrices. 


(a) How many members belong to the groups of 4 by 4 and n by n permutation 
matrices? 


(b) Find a power k so that all 3 by 3 permutation matrices satisfy P* = /. 
Describe the rows of DA and the columns of AD if D = [3 9]. 
(a) If A is invertible what is the inverse of A'? 


(b) If A is also symmetric what is the transpose of AnD 


(c) Illustrate both formulas when A = [7 1]. 


By experiment with n = 2 and n = 3, find 


Starting with a first plane u + 2v — w = 6, find the equation for 


(a) the parallel plane through the origin. 
(b) a second plane that also contains the points (6,0,0) and (2,2,0). 
(c) a third plane that meets the first and second in the point (4, 1,0). 


What multiple of row 2 is subtracted from row 3 in forward elimination of A? 


120 
0; }O 1 5 
1} |0 0 1 

How do you know (without multiplying those factors) that A is invertible, symmet- 


ric, and tridiagonal? What are its pivots? 


(a) What vector x will make Ax = column 1 of A + 2(column 3), for a 3 by 3 matrix 
A? 
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(b) Construct a matrix that has column 1 + 2(column 3) = 0. Check that A is 
singular (fewer than 3 pivots) and explain why that must be the case. 
1.27 True or false, with reason if true and counterexample if false: 


(1) If L;U, = LyU>2 (upper triangular U’s with nonzero diagonal, lower triangular 
L’s with unit diagonal), then L; = Lz and U; = U2. The LU factorization is 
unique. 


(2) IfA*+A =/ then A~! =A+4+1. 


(3) If all diagonal entries of A are zero, then A is singular. 


1.28 By experiment or the Gauss-Jordan method compute 


n —l —1 


1 
id 
m 


or Oo 
r- Oo SO 


1 00 1 
‘ 210 ; L 
m0 1 0 


1.29 Write down the 2 by 2 matrices that 


(a) reverse the direction of every vector. 
(b) project every vector onto the x2 axis. 
(c) turn every vector counterclockwise through 90°. 


(d) reflect every vector through the 45° line x; = x2. 


Vector Spaces 


2.1 Vector Spaces and Subspaces 


Elimination can simplify, one entry at a time, the linear system Ax = b. Fortunately it 
also simplifies the theory. The basic questions of existence and uniqueness—Is there 
one solution, or no solution, or an infinity of solutions?—are much easier to answer 
after elimination, We need to devote one more section to those questions, to find every 
solution for an m by n system. Then that circle of ideas will be complete. 

But elimination produces only one kind of understanding of Ax = b. Our chief object 
is to achieve a different and deeper understanding. This chapter may be more difficult 
than the first one. It goes to the heart of linear algebra. 

For the concept of a vector space, we start immediately with the most important 
spaces. They are denoted by R!,R’,R?,...; the space R” consists of all column vectors 
with n components. (We write R because the components are real numbers.) R? is 
represented by the usual x-y plane; the two components of the vector become the x and 
y coordinates of the corresponding point. The three components of a vector in R? give a 
point in three-dimensional space. The one-dimensional space R! is a line. 

The valuable thing for linear algebra is that the extension to n dimensions is so 
straightforward. For a vector in R’ we just need the seven components, even if the 
geometry is hard to visualize. Within all vector spaces, two operations are possible: 


We can add any two vectors, and we can multiply all vectors by scalars. 
In other words, we can take linear combinations. 


Addition obeys the commutative law x + y = y+ x; there is a “zero vector” satisfying 
0+ x =x; and there is a vector “—x” satisfying —x +x = 0. Eight properties (including 
those three) are fundamental; the full list is given in Problem 5 at the end of this section. 
A real vector space is a set of vectors together with rules for vector addition and mul- 
tiplication by real numbers. Addition and multiplication must produce vectors in the 
space, and they must satisfy the eight conditions. 
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Normally our vectors belong to one of the spaces R”; they are ordinary column vec- 
tors. If x = (1,0,0,3), then 2x (and also x +x) has components 2, 0, 0, 6. The formal 
definition allows other things to be “vectors”-provided that addition and scalar multipli- 
cation are all right. We give three examples: 


1. The infinite-dimensional space R®. Its vectors have infinitely many components, as 
in x = (1,2,1,2,...). The laws for x+ y and cx stay unchanged. 


2. The space of 3 by 2 matrices. In this case the “vectors” are matrices! We can add 
two matrices, and A+ B = B+A, and there is a zero matrix, and so on. This space 
is almost the same as R°. (The six components are arranged in a rectangle instead 
of acolumn.) Any choice of m and n would give, as a similar example, the vector 
space of all m by n matrices. 


3. The space of functions f(x). Here we admit all functions f that are defined on 
a fixed interval, say 0 <x < 1. The space includes f(x) = x”, g(x) = sinx, their 
sum (f + g)(x) =x? +sin.x, and all multiples like 3x7 and — sinx. The vectors are 
functions, and the dimension is somehow a larger infinity than for R™. 


Other examples are given in the exercises, but the vector spaces we need most are 
somewhere else—they are inside the standard spaces R". We want to describe them 
and explain why they are important. Geometrically, think of the usual three-dimensional 
R? and choose any plane through the origin. That plane is a vector space in its own 
right. If we multiply a vector in the plane by 3, or —3, or any other scalar, we get a 
vector in the same plane. If we add two vectors in the plane, their sum stays in the 
plane. This plane through (0,0,0) illustrates one of the most fundamental ideas in linear 
algebra; it is a subspace of the original space R°. 


Definition. A subspace of a vector space is a nonempty subset that satisfies the require- 
ments for a vector space: Linear combinations stay in the subspace. 


(i) If we add any vectors x and y in the subspace, x + y is in the subspace. 


(ii) If we multiply any vector x in the subspace by any scalar c, cx is in the subspace. 


Notice our emphasis on the word space. A subspace is a subset that is “closed” under 
addition and scalar multiplication. Those operations follow the rules of the host space, 
keeping us inside the subspace. The eight required properties are satisfied in the larger 
space and will automatically be satisfied in every subspace. Notice in particular that the 
zero vector will belong to every subspace. That comes from rule (ii): Choose the scalar 
to bec = 0. 

The smallest subspace Z contains only one vector, the zero vector. It is a “zero- 
dimensional space,” containing only the point at the origin. Rules (1) and (11) are satisfied, 
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since the sum 0+ 0 is in this one-point space, and so are all multiples cO. This is the 
smallest possible vector space: the empty set is not allowed. At the other extreme. the 
largest subspace is the whole of the original space. If the original space is R*, then the 
possible subspaces are easy to describe: R? itself, any plane through the origin, any line 
through the origin, or the origin (the zero vector) alone. 

The distinction between a subset and a subspace is made clear by examples. In each 
case, can you add vectors and multiply by scalars, without leaving the space? 


Example 1. Consider all vectors in R? whose components are positive or zero. This 
subset is the first quadrant of the x-y plane; the coordinates satisfy x > 0 and y > 0. It 
is not a subspace, even though it contains zero and addition does leave us within the 
subset. Rule (ii) is violated, since if the scalar is —1 and the vector is [1 1], the multiple 
cx =|—1 —1] is in the third quadrant instead of the first. 


If we include the third quadrant along with the first, scalar multiplication is all right. 
Every multiple cx will stay in this subset. However, rule (i) is now violated, since adding 
[1 2}+[-2 —1] gives [—1 1], which is not in either quadrant. The smallest subspace 
containing the first quadrant is the whole space R?. 


Example 2. Start from the vector space of 3 by 3 matrices. One possible subspace is 
the set of lower triangular matrices. Another is the set of symmetric matrices. A+B 
and cA are lower triangular if A and B are lower triangular, and they are symmetric if A 
and B are symmetric. Of course, the zero matrix is in both subspaces. 


The Column Space of A 


We now come to the key examples, the column space and the nullspace of a matrix 
A. The column space contains all linear combinations of the columns of A. It is a 
subspace of R”. We illustrate by a system of m = 3 equations in n = 2 unknowns: 


1 0 by 
Combination of columns equals b 5 4 a = |b2/. (1) 
2 4 i b3 
With m > n we have more equations than unknowns—and usually there will be no solu- 


tion. The system will be solvable only for a very “thin” subset of all possible b’s. One 
way of describing this thin subset is so simple that it is easy to overlook. 


2A The system Ax = bis solvable if and only if the vector b can be expressed 
as a combination of the columns of A. Then b is in the column space. 


This description involves nothing more than a restatement of Ax = b, by columns: 


1 0 b, 
Combination of columns ul5|+vi4) = |bo|. (2) 
2 4 b3 
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perpendicular 
to plane 


column 1 


column space 


Figure 2.1: The column space C(A), a plane in three-dimensional space. 


These are the same three equations in two unknowns. Now the problem is: Find numbers 
u and v that multiply the first and second columns to produce b. The system is solvable 
exactly when such coefficients exist, and the vector (u,v) is the solution x. 

We are saying that the attainable right-hand sides b are all combinations of the columns 
of A. One possible right-hand side is the first column itself; the weights are uv = 1 and 
v =0. Another possibility is the second column: u = 0 and v = 1. A third is the right- 
hand side b = 0. With u = 0 and v = 0, the vector b = 0 will always be attainable. 

We can describe all combinations of the two columns geometrically: Ax = b can be 
solved if and only if b lies in the plane that is spanned by the two column vectors (Figure 
2.1). This is the thin set of attainable b. If b lies off the plane, then it is not a combination 
of the two columns. In that case Ax = b has no solution. 

What is important is that this plane is not just a subset of R° it is a subspace. It is 
the column space of A, consisting of all combinations of the columns. It is denoted by 
C(A). Requirements (i) and (ii) for a subspace of R” are easy to check: 


(i) Suppose b and b’ lie in the column space, so that Ax = b for some x and Ax’ = b’ 
for some x’. Then A(x+x’) =b+D’, so that b+’ is also a combination of the 
columns. The column space of all attainable vectors b is closed under addition. 


(ii) If b is in the column space C(A), so is any multiple cb. If some combination 
of columns produces b (say Ax = b), then multiplying that combination by c will 
produce cb. In other words, A(cx) = cb. 


For another matrix A, the dimensions in Figure 2.1 may be very different. The small- 
est possible column space (one vector only) comes from the zero matrix A = 0. The 
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only combination of the columns is b = 0. At the other extreme, suppose A is the 5 by 
5 identity matrix. Then C((/) is the whole of R°; the five columns of J can combine to 
produce any five-dimensional vector b. This is not at all special to the identity matrix. 
Any 5 by 5 matrix that is nonsingular will have the whole of R° as its column space. 
For such a matrix we can solve Ax = b by Gaussian elimination; there are five pivots. 
Therefore every b is in C(A) for a nonsingular matrix. 

You can see how Chapter | is contained in this chapter. There we studied n by n 
matrices whose column space is R”. Now we allow singular matrices, and rectangu- 
lar matrices of any shape. Then C(A) can be somewhere between the zero space and 
the whole space R”. Together with its perpendicular space, it gives one of our two 
approaches to understanding Ax = b. 


The Nullspace of A 


The second approach to Ax = b is “dual” to the first. We are concerned not only with 
attainable right-hand sides b, but also with the solutions x that attain them. The right- 
hand side b = 0 always allows the solution x = 0, but there may be infinitely many other 
solutions. (There always are, if there are more unknowns than equations, n > m.) The 
solutions to Ax = 0 form a vector space—the nullspace of A. 


The nullspace of a matrix consists of all vectors x such that Ax = 0. It is 
denoted by N(A). It is a subspace of R”, just as the column space was a 
subspace of R”. 


Requirement (i) holds: If Ax = 0 and Ax’ = 0, then A(x +’) = 0. Requirement (ii) also 
holds: If Ax = 0 then A(cx) = 0. Both requirements fail if the right-hand side is not zero! 
Only the solutions to a homogeneous equation (b = 0) form a subspace. The nullspace 
is easy to find for the example given above; it is as small as possible: 


10 0 
5 4) "] =o 
a 0 


The first equation gives u = 0, and the second equation then forces v = 0. The nullspace 
contains only the vector (0,0). This matrix has “independent columns”—a key idea that 
comes soon. 

The situation is changed when a third column is a combination of the first two: 


101 
Larger nullspace B= |5 4 9 
2 4 6 


B has the same column space as A. The new column lies in the plane of Figure 2.1; it is 
the sum of the two column vectors we started with. But the nullspace of B contains the 
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vector (1,1,—1) and automatically contains any multiple (c,c,—c): 


101 0 
Nullspace is a line 5 4 9 E G -<| = |0 
2 4 6 0 


The nullspace of B is the line of all points x = c, y=c, z = —c. (The line goes through 
the origin, as any subspace must.) We want to be able, for any system Ax = J, to find 
C(A) and N(A): all attainable right-hand sides b and all solutions to Ax = 0. 

The vectors b are in the column space and the vectors x are in the nullspace. We shall 
compute the dimensions of those subspaces and a convenient set of vectors to generate 
them. We hope to end up by understanding all four of the subspaces that are intimately 
related to each other and to A—the column space of A, the nullspace of A, and their two 
perpendicular spaces. 


Problem Set 2.1 


1. Construct a subset of the x-y plane R? that is 


(a) closed under vector addition and subtraction, but not scalar multiplication. 


(b) closed under scalar multiplication but not under vector addition. 
Hint: Starting with u and v, add and subtract for (a). Try cu and cv for (b). 
2. Which of the following subsets of R? are actually subspaces? 


(a) The plane of vectors (b,b2,b3) with first component b; = 0. 
(b) The plane of vectors b with bj = 1. 


(c) The vectors b with b2b3 = 0 (this is the union of two subspaces, the plane bz = 0 
and the plane b3 = 0). 


(d) All combinations of two given vectors (1,1,0) and (2,0, 1). 
(e) The plane of vectors (b;,b2,b3) that satisfy b3 — bz + 3b; = 0. 


3. Describe the column space and the nullspace of the matrices 


A= : Fi and =| : ; and C= 


0 203 0 0 0 


toe 


4. What is the smallest subspace of 3 by 3 matrices that contains all symmetric matrices 
and all lower triangular matrices? What is the largest subspace that is contained in 
both of those subspaces? 


5. Addition and scalar multiplication are required to satisfy these eight rules: 
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xX+y=yt+x. 

x+(y+z) = (x+y) +z. 

There is a unique “zero vector” such that x + 0 = x for all x. 
For each x there is a unique vector —x such that x+ (—x) = 0. 
be= 7: 

(cjc2)x = c1(C2x). 


c(xt+ty) =cx+cy. 


COs sO Sa a ee 


(cy +02)x = cyx+ Cox. 


(a) Suppose addition in R? adds an extra 1 to each component, so that (3, 1) + (5,0) 
equals (9,2) instead of (8,1). With scalar multiplication unchanged, which rules 
are broken? 


(b) Show that the set of all positive real numbers, with x+ y and cx redefined to equal 
the usual xy and x°, is a vector space. What is the “zero vector’? 


(c) Suppose (x1,x2) + (91,2) is defined to be (x; +y2,x2+y,). With the usual 
CX = (cxX1,CX2), which of the eight conditions are not satisfied? 


. Let P be the plane in 3-space with equation x + 2y+ z= 6. What is the equation of 
the plane Po through the origin parallel to P? Are P and Po subspaces of R*? 


. Which of the following are subspaces of R”? 

(a) All sequences like (1,0,1,0,...) that include infinitely many zeros. 
(b) All sequences (x1,x2,...) with x; = 0 from some point onward. 

(c) All decreasing sequences: xj41 <x; for each j. 

(d) All convergent sequences: the x; have a limit as j — ©. 

(e) All arithmetic progressions: xj; —xj; is the same for all j. 


(f) All geometric progressions (x1, kx1,k7x1,...) allowing all k and x1. 


. Which of the following descriptions are correct? The solutions x of 


ai 
111 0 
Ax = OM 
102 0 
X3 


form 

(a) a plane. 
(b) a line. 
(c) a point. 


(d) a subspace. 
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10. 


11. 


12. 


13. 


14. 


15. 


16. 


17. 


18. 
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(e) the nullspace of A. 
(f) the column space of A. 


. Show that the set of nonsingular 2 by 2 matrices is not a vector space. Show also that 


the set of singular 2 by 2 matrices is not a vector space. 


The matrix A = [5 —3] is a “vector” in the space M of all 2 by 2 matrices. Write the 
zero vector in this space, the vector 5A, and the vector —A. What matrices are in the 
smallest subspace containing A? 

(a) Describe a subspace of M that contains A = |} }}| but not B= [9 ©, ]. 

(b) If a subspace of M contains A and B, must it contain /? 

(c) Describe a subspace of M that contains no nonzero diagonal matrices. 

The functions f(x) =x? and g(x) = 5x are “vectors” in the vector space F of all real 


functions. The combination 3 f(x) — 4g(x) is the function h(x) = . Which rule 
is broken if multiplying f(x) by c gives the function f(cx)? 


If the sum of the “vectors” f(x) and g(x) in F is defined to be f(g(x)), then the “zero 
vector” is g(x) =x. Keep the usual scalar multiplication cf(x), and find two rules 
that are broken. 


Describe the smallest subspace of the 2 by 2 matrix space M that contains 


(a) F ; and ; f (b) F ; and F | 

0 0 0 0 0 0 01 

1 1 1o] fo. 
o F ‘ @) k Ar : F ; iF 


Let P be the plane in R° with equation x + y— 2z = 4. The origin (0,0,0) is not in 
P! Find two vectors in P and check that their sum is not in P. 


Po is the plane through (0,0,0) parallel to the plane P in Problem 15. What is the 
equation for Po? Find two vectors in Pop and check that their sum is in Po. 


The four types of subspaces of R? are planes, lines, R? itself, or Z containing only 

(0,0,0). 

(a) Describe the three types of subspaces of R?. 

(b) Describe the five types of subspaces of R*. 

(a) The intersection of two planes through (0,0,0) is probably a but it could 
bea . It can’t be the zero vector Z! 

(b) The intersection of a plane through (0,0,0) with a line through (0,0,0) is prob- 
ably a but it could be a ‘ 


19. 


20. 


21. 


22. 


23. 


24. 


25. 


26. 
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(c) If S and T are subspaces of R>, their intersection SMT (vectors in both sub- 
spaces) is a subspace of R>. Check the requirements on x + y and cx. 


Suppose P is a plane through (0,0,0) and L is a line through (0,0,0). The smallest 
vector space containing both P and L is either or 


True or false for M = all 3 by 3 matrices (check addition using an example)? 


(a) The skew-symmetric matrices in M (with A! = —A) forma subspace. 
(b) The unsymmetric matrices in M (with AY + A) form a subspace. 


(c) The matrices that have (1,1, 1) in their nullspace form a subspace. 
Problems 21-30 are about column spaces C(A) and the equation Ax = b. 


Describe the column spaces (lines or planes) of these particular matrices: 


| oe. 1 O 1 0 
A=10 0 and B=|0 2 and. -C=)'2° 0 
0 0 0 0 0 0 


For which right-hand sides (find a condition on b, b2, b3) are these systems solvable? 


1 4 2 X{ by 1 4 bi 
@) |2 8 4/Iml=|ml. © |2 9 => 
—1 -4 -2 X3 b3 —1 —4 ‘ b3 
Adding row 1 of A to row 2 produces B. Adding column | to column 2 produces C. 
A combination of the columns of ____ is also a combination of the columns of A. 
Which two matrices have the same column __s? 


ae 


A) hed PSle ol wade Sl 
2 4 3 6 2 6 


For which vectors (b;,b2,b3) do these systems have a solution? 


11 1] |x by Lol ly jx by 
O 1 1} [xo] = |b and O 1 1} |x| = |b 
00 1 X3 b3 00 0 X3 bz 


(Recommended) If we add an extra column b to a matrix A, then the column space 
gets larger unless ___. Give an example in which the column space gets larger and 
an example in which it doesn’t. Why is Ax = b solvable exactly when the column 
space doesn’t get larger by including b? 


The columns of AB are combinations of the columns of A. This means: The column 
space of AB is contained in (possibly equal to) the column space of A. Give an 
example where the column spaces of A and AB are not equal. 
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27. If A is any 8 by 8 invertible matrix, then its column space is . Why? 
28. True or false (with a counterexample if false)? 


(a) The vectors b that are not in the column space C(A) form a subspace. 
(b) If C(A) contains only the zero vector, then A is the zero matrix. 
(c) The column space of 2A equals the column space of A. 


(d) The column space of A — J equals the column space of A. 


29. Construct a 3 by 3 matrix whose column space contains (1, 1,0) and (1,0, 1) but not 
(1,1,1). Construct a 3 by 3 matrix whose column space is only a line. 


30. Ifthe 9 by 12 system Ax = b is solvable for every b, then C(A) = 
31. Why isn’t R? a subspace of R*? 


2.2 Solving Ax = 0 and Ax = b 


Chapter 1 concentrated on square invertible matrices. There was one solution to Ax = b 
and it was x = —A~!b. That solution was found by elimination (not by computing A~!). 
A rectangular matrix brings new possibilities—U may not have a full set of pivots. This 
section goes onward from U to a reduced form R—the simplest matrix that elimina- 
tion can give. R reveals all solutions immediately. 

For an invertible matrix, the nullspace contains only x = 0 (multiply Ax = 0 by A~!). 
The column space is the whole space (Ax = b has a solution for every b). The new ques- 
tions appear when the nullspace contains more than the zero vector and/or the column 
space contains Jess than all vectors: 


1. Any vector x, in the nullspace can be added to a particular solution x,. The solutions 
to all linear equations have this form, x = Xp + Xn: 


Complete solution Axp=b and Ax, =0 produce A(x,+xXn) =b. 


2. When the column space doesn’t contain every b in R”, we need the conditions on 
b that make Ax = b solvable. 


A 3 by 4 example will be a good size. We will write down all solutions to Ax = 0. We 
will find the conditions for b to lie in the column space (so that Ax = b is solvable). The 
1 by 1 system Ox = b, one equation and one unknown, shows two possibilities: 


Ox = b has no solution unless b = 0. The column space of the 1 by 1 zero 
matrix contains only b= 0. 


Ox = 0 has infinitely many solutions. The nullspace contains all x. A particular 
solution is x,» = 0, and the complete solution is x = x) +.Xn = 0+ (any x). 
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Simple, I admit. If you move up to 2 by 2, it’s more interesting. The matrix E 5 | is not 
invertible: y+ z= b; and 2y + 2z = b2 usually have no solution. 


There is no solution unless by = 2b;. The column space of A contains only 
those b’s, the multiples of (1,2). 

When bz = 2b, there are infinitely many solutions. A particular solution to 
y+z=2 and 2y+2z = 4 is x» = (1,1). The nullspace of A in Figure 2.2 
contains (—1,1) and all its multiples x, = (—c,c): 


Complete yt 2. = 


is solved by x,+x%),= 
solution aye 22:= ee 


nullspace Ax, = 0 


Figure 2.2: The parallel lines of solutions to Ax, = 0 and Ee A | PR} = Fal 


Echelon Form U and Row Reduced Form R 


We start by simplifying this 3 by 4 matrix, first to U and then further to R: 


i 2 22 
Basic example A=|2 697 
—-1 -3 3 4 


The pivot a}; = 1 is nonzero. The usual elementary operations will produce zeros in the 
first column below this pivot. The bad news appears in column 2: 


iro 3 2 
No pivot in column 2 A—|0 0 3 3 
00 6 6 


The candidate for the second pivot has become zero: unacceptable. We look below that 
zero for a nonzero entry—intending to carry out a row exchange. In this case the entry 
below it is also zero. If A were square, this would signal that the matrix was singular. 
With a rectangular matrix, we must expect trouble anyway, and there is no reason to stop. 
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All we can do is to go on to the next column, where the pivot entry is 3. Subtracting twice 
the second row from the third, we arrive at U: 


Echelon matrix U U= 


oo = 
oOo WwW 
Oo & WwW 
oOW WN 


Strictly speaking, we proceed to the fourth column. A zero is in the third pivot position, 
and nothing can be done. U is upper triangular, but its pivots are not on the main diago- 
nal. The nonzero entries of U have a “staircase pattern,” or echelon form. For the 5 by 
8 case in Figure 2.3, the starred entries may or may not be zero. 


ek kK KK OK OK OK 10 * O « x « O 
Cle ee 4-2 @ 2 Ode Ose ee 20 
U=|]0 0 Oje « «x «x «x R=|0 0 0]1 *« «x « 0 
0000 0 0 Oje 00000 0 0/1 
00000000 00000000 


Figure 2.3: The entries of a 5 by 8 echelon matrix U and its reduced form R. 


We can always reach this echelon form U, with zeros below the pivots: 
1. The pivots are the first nonzero entries in their rows. 
2. Below each pivot is a column of zeros, obtained by elimination. 


3. Each pivot lies to the right of the pivot in the row above. This produces the staircase 
pattern, and zero rows come last. 


Since we started with A and ended with U, the reader is certain to ask: Do we have 
A = LU as before? There is no reason why not, since the elimination steps have not 
changed. Each step still subtracts a multiple of one row from a row beneath it. The 
inverse of each step adds back the multiple that was subtracted. These inverses come in 
the right order to put the multipliers directly into L: 


1 0 0 
Lower triangular BS 2. lO and A=LWU. 
-1 21 


Note that L is square. It has the same number of rows as A and U. 

The only operation not required by our example, but needed in general, is row ex- 
change by a permutation matrix P. Since we keep going to the next column when no 
pivots are available, there is no need to assume that A is nonsingular. Here is PA = LU 
for all matrices: 


2.2 Solving Ax = 0 and Ax = b 89 


2B For any m by n matrix A there is a permutation P, a lower triangular L 
with unit diagonal, and an m by n echelon matrix U, such that PA = LU. 


Now comes R. We can go further than U, to make the matrix even simpler. Divide 
the second row by its pivot 3, so that all pivots are 1. Then use the pivot row to produce 
zero above the pivot. This time we subtract a row from a higher row. The final result 
(the best form we can get) is the reduced row echelon form R: 


fs 3:2 | Dales eo ae 13 0 -1 
00 3 3} —|001 1} —]0 01 1|]=R. 
000 0 00 0 0 000 0 


This matrix R is the final result of elimination on A. MATLAB would use the command 
R = rref(A). Of course rref(R) would give R again! 

What is the row reduced form of a square invertible matrix? In that case R is the 
identity matrix. There is a full set of pivots, all equal to 1, with zeros above and below. 
So rref(A) = I, when A is invertible. 

For a 5 by 8 matrix with four pivots, Figure 2.3 shows the reduced form R. It still 
contains an identity matrix, in the four pivot rows and four pivot columns. From R 
we will quickly find the nullspace of A. Rx = 0 has the same solutions as Ux = 0 and 
Ax = 0. 


Pivot Variables and Free Variables 


Our goal is to read off all the solutions to Rx = 0. The pivots are crucial: 


Nullspace of R 13 0 -1 . 0 
v 

(pivot columns kx=|001 1 = |0 
Ww 

in boldface) 000 O 0 
, 


The unknowns u, v, w, y go into two groups. One group contains the pivot variables, 
those that correspond to columns with pivots. The first and third columns contain the 
pivots, so u and w are the pivot variables. The other group is made up of the free 
variables, corresponding to columns without pivots. These are the second and fourth 
columns, so v and y are free variables. 

To find the most general solution to Rx = 0 (or, equivalently, to Ax = 0) we may assign 
arbitrary values to the free variables. Suppose we call these values simply v and y. The 
pivot variables are completely determined in terms of v and y: 


ut+3v—y=0 yields u=-—3v-+y 


Rx =0 (1) 


wt+y=0 _ yields w= —y 
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There is a “double infinity” of solutions, with v and y free and independent. The com- 
plete solution is a combination of two special solutions: 


. —3v+y —3 1 
Nullspace contains 1 0 
v 
all combinations i =v 0 +y 1 (2) 
of special solutions : 
y 0 1 


Please look again at this complete solution to Rx = 0 and Ax = 0. The special solution 
(—3,1,0,0) has free variables v = 1, y = 0. The other special solution (1,0,—1,1) has 
v =O and y= 1. All solutions are linear combinations of these two. The best way to find 
all solutions to Ax = 0 is from the special solutions: 


1. After reaching Rx = 0, identify the pivot variables and free variables. 


2. Give one free variable the value 1, set the other free variables to 0, and solve Rx = 0 
for the pivot variables. This x is a special solution. 


3. Every free variable produces its own “special solution” by step 2. The combinations 
of special solutions form the nullspace—all solutions to Ax = 0. 


Within the four-dimensional space of all possible vectors x, the solutions to Ax = 0 
form a two-dimensional subspace—the nullspace of A, In the example, N(A) is gener- 
ated by the special vectors (—3,1,0,0) and (1,0,—1,1). The combinations of these two 
vectors produce the whole nullspace. 

Here is a little trick. The special solutions are especially easy from R. The numbers 3 
and 0 and —1 and 1 lie in the “nonpivot columns” of R. Reverse their signs to find the 
pivot variables (not free) in the special solutions. I will put the two special solutions 
from equation (2) into a nullspace matrix N, so you see this neat pattern: 


Null tri -3 1 not free 
ullspace matrix - a . 
(columns are N= 

special solutions) 0 -1 not free 

° : 0 1 free 


The free variables have values 1 and 0. When the free columns moved to the right- 
hand side of equation (2), their coefficients 3 and 0 and —1 and | switched sign. That 
determined the pivot variables in the special solutions (the columns of N). 

This is the place to recognize one extremely important theorem. Suppose a matrix has 
more columns than rows, 1 > m. Since m rows can hold at most m pivots, there must be 
at least n — m free variables. There will be even more free variables if some rows of R 
reduce to zero; but no matter what, at least one variable must be free. This free variable 
can be assigned any value, leading to the following conclusion: 


2C If Ax =0 has more unknowns than equations (n > m), it has at least one 
special solution: There are more solutions than the trivial x = 0. 
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There must be infinitely many solutions, since any multiple cx will also satisfy A(cx) = 
0. The nullspace contains the line through x. And if there are additional free variables, 
the nullspace becomes more than just a line in n-dimensional space. The nullspace has 
the same “dimension” as the number of free variables and special solutions. 

This central idea—the dimension of a subspace—is made precise in the next section. 
We count the free variables for the nullspace. We count the pivot variables for the column 
space! 


Solving Ax = b, Ux =c, and Rx =d 


The case b £ 0 is quite different from b = 0. The row operations on A must act also 
on the right-hand side (on b). We begin with letters (b,b2,b3) to find the solvability 
condition—for b to lie in the column space. Then we choose b = (1,5,5) and find all 
solutions x. 

For the original example Ax = b = (b;,b2,b3), apply to both sides the operations that 
led from A to U. The result is an upper triangular system Ux = c: 


age ae by 
Vv 
Ux=c 00 3 3 = by — 2b, . (3) 
Ww 
0000 b3 — 2b. + 5b, 


The vector c on the right-hand side, which appeared after the forward elimination steps, 
is just L~'D as in the previous chapter. Start now with Ux = c. 

It is not clear that these equations have a solution. The third equation is very much 
in doubt, because its left-hand side is zero. The equations are inconsistent unless b3 — 
2b7 + 5b; = 0. Even though there are more unknowns than equations, there may be no 
solution. We know another way of answering the same question: Ax = b can be solved if 
and only if b lies in the column space of A. This subspace comes from the four columns 
of A (not of U!): 


Columns of A 1 3 3 2 
“span” the Saal 614 9] , o 
column space —1 —3 3 4 


Even though there are four vectors, their combinations only fill out a plane in three- 
dimensional space. Column 2 is three times column 1. The fourth column equals the 
third minus the first. These dependent columns, the second and fourth, are exactly the 
ones without pivots. 

The column space C'(A) can be described in two different ways. On the one hand, 
it is the plane generated by columns I and 3. The other columns lie in that plane, 
and contribute nothing new. Equivalently, it is the plane of all vectors b that satisfy 
b3 — 2b + 5b; = 0; this is the constraint if the system is to be solvable. Every column 
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satisfies this constraint, so it is forced on b! Geometrically, we shall see that the vector 
(5, —2,1) is perpendicular to each column. 

If b belongs to the column space, the solutions of Ax = b are easy to find. The last 
equation in Ux = c is 0 =0. To the free variables v and y, we may assign any values, 
as before. The pivot variables u and w are still determined by back-substitution. For a 
specific example with b3 — 2b7 + 5b; = 0, choose b = (1,5,5): 


SV Ss oS 
II 
nn Oe 


< 


Se 


1 3 
UX=<¢ 0 0 
0 0 


| 
So WO ke 


y 


The last equation is 0 = 0, as expected. Back-substitution gives 


3w+3y=3 or w=l1-y 
u+3v+3w+2y=1 or u=—2—3yv-+y¥. 


Again there is a double infinity of solutions: v and y are free, u and w are not: 


le -2 —3 1 

Complete solution v 0 1 0 

y ae ae ee +y +y (4) 
y 0 0 1 

This has all solutions to Ax = 0, plus the new x, = (—2,0,1,0). That x, is a particular 


solution to Ax = b. The last two terms with v and y yield more solutions (because they 
satisfy Ax = 0). Every solution to Ax = b is the sum of one particular solution and a 
solution to Ax = 0: 


Xcomplete = “particular + Xnullspace 


The particular solution in equation (4) comes from solving the equation with all free 
variables set to zero. That is the only new part, since the nullspace is already computed. 
When you multiply the highlighted equation by A, you get Axcomplete = D + 0. 

Geometrically, the solutions again fill a two-dimensional surface—but it is not a sub- 
space. It does not contain x = 0. It is parallel to the nullspace we had before, shifted 
by the particular solution x, as in Figure 2.2. Equation (4) is a good way to write the 
answer: 


1. Reduce Ax = b to Ux =c. 
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2. With free variables = 0, find a particular solution to Ax, = b and Uxp =. 


3. Find the special solutions to Ax = 0 (or Ux = 0 or Rx = 0). Each free variable, in 
turn, is 1. Then x = x, + (any combination x, of special solutions). 


When the equation was Ax = 0, the particular solution was the zero vector! It fits the pat- 
tern, but Xparticular = 0 was not written in equation (2). Now x, is added to the nullspace 
solutions, as in equation (4). 

Question: How does the reduced form R make this solution even clearer? You will 
see it in our example. Subtract equation 2 from equation 1, and then divide equation 2 
by its pivot. On the left-hand side, this produces R, as before. On the right-hand side, 
these operations change c = (1,3,0) to a new vector d = (—2, 1,0): 


Reduced equation bea =e 
u uati v 
hy? 0011 =]1|. (5) 
ian 000 0 "| 
y 


Our particular solution x,, (one choice out of many) has free variables v = y = 0. 
Columns 2 and 4 can be ignored. Then we immediately have u = —2 and w = 1, exactly 
as in equation (4). The entries of d go directly into x,. This is because the identity 
matrix is sitting in the pivot columns of R! 

Let me summarize this section, before working a new example. Elimination reveals 
the pivot variables and free variables. If there are r pivots, there are r pivot variables 
and n —r free variables. That important number r will be given a name—it is the rank 
of the matrix. 


2D Suppose elimination reduces Ax = b to Ux = c and Rx = d, with r pivot 
rows and r pivot columns. The rank of those matrices is r. The last m—r 
rows of U and R are zero, so there is a solution only if the last m— r entries of 
c and d are also zero. 


The complete solution is x = x») +Xn. One particular solution x, has all free 
variables zero. Its pivot variables are the first r entries of d, so Rx» = d. 


The nullspace solutions x, are combinations of n —r special solutions, with 
one free variable equal to 1. The pivot variables in that special solution can be 
found in the corresponding column of R (with sign reversed). 


You see how the rank r is crucial. It counts the pivot rows in the “row space” and the 
pivot columns in the column space. There are n — r special solutions in the nullspace. 
There are m—r solvability conditions on b orc or d. 
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Another Worked Example 


The full picture uses elimination and pivot columns to find the column space, nullspace, 
and rank. The 3 by 4 matrix A has rank 2: 


Ix, + 2x%. + 3x43 + 5x, = JD 


Ax=b is 2x, + 40 + 8x3 4+ 12% = Dn (6) 
3x, + 6x. + 7x3 + 13x4 = bz 
1. Reduce [A }] to [U c], to reach a triangular system Ux = c. 
2. Find the condition on 51, b2, b3 to have a solution. 
3. Describe the column space of A: Which plane in R*? 
4. Describe the nullspace of A: Which special solutions in R*? 
5. Find a particular solution to Ax = (0,6, —6) and the complete x, + xp. 
6. Reduce [U c] to [R d]: Special solutions from R and x, from d. 
Solution. (Notice how the right-hand side is included as an extra column!) 
1. The multipliers in elimination are 2 and 3 and —1, taking [A }] to [U c]. 
| ees Se | U2 3 De |) Oa 123 5\b 
24 8 12 b2 |}—|00 2 2 | bg—2b; | —| 0 0 2 2) bg— 2b, 
3 6 7 13 bg 0 0 —2 —2| b3—3b, 0 0 0 0| b3+b2—5b4 


2. The last equation shows the solvability condition b3 + by —5b, = 0. Then 0 = 0. 


3. The column space of A is the plane containing all combinations of the pivot columns 
(1,2,3) and (3,8,7). 
Second description: The column space contains all vectors with b3 + bz — 5b; = 0. 
That makes Ax = b solvable, so b is in the column space. All columns of A pass this 
test b3 + bz —5b, = 0. This is the equation for the plane (in the first description of 
the column space). 


4. The special solutions in N have free variables x2 = 1, x4 = 0 and x2 = 0, x4 = 1: 


Nullspace matrix —2 —2 
Special solutions to Ax = 0 N= 1 O 
Back-substitution in Ux = 0 NSO. ea 


Just switch signs in Rx = 0 0 1 
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5. Choose b = (0,6, —6), which has b3 + bz — 5b, = 0. Elimination takes Ax = b to 
Ux =c = (0,6,0). Back-substitute with free variables = 0: 


=9 
Oral st 
Particular solution to Ax, = (0,6, —6) oa a 
0 | free 


The complete solution to Ax = (0,6, —6) is (this x,) + (all x;). 


6. In the reduced R, the third column changes from (3,2,0) to (0, 1,0). The right-hand 
side c = (0,6,0) becomes d = (—9,3,0). Then —9 and 3 go into x,: 


5/0 12 
26|— |r al=|0 0 
0 0 0 0 


That final matrix [R d] is rref([A b]) =rref([U c]). The numbers 2 and 0 and 2 
and 1 in the free columns of R have opposite sign in the special solutions (the nullspace 
matrix NV). Everything is revealed by Rx = d. 


Problem Set 2.2 


1. Construct a system with more unknowns than equations, but no solution. Change the 
right-hand side to zero and find all solutions xy. 


2. Reduce A and B to echelon form, to find their ranks. Which variables are free? 


Find the special solutions to Ax = 0 and Bx = 0. Find all solutions. 
3. Find the echelon form U, the free variables, and the special solutions: 
010 3 i by 
020 6’ | bo] 
Ax = b is consistent (has a solution) when b satisfies by = . Find the complete 
solution in the same form as equation (4). 


hea 


96 


10. 
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. Carry out the same steps as in the previous problem to find the complete solution of 


Mx=b: 
0 O by 
1 2 
M= . b= bo 
0 0 b3 
3 6 ba 


. Write the complete solutions x = x, +X, to these systems, as in equation (4): 


Uu Uu 
Lod 2 1 Zag 1 
viz vj = . 
245 4 244 4 
WwW WwW 


Describe the set of attainable right-hand sides b (in the column space) for 


1 0 ‘ bi 
01 a mI 
278 , b3 


by finding the constraints on b that turn the third equation into 0 = 0 (after elimina- 
tion). What is the rank, and a particular solution? 


. Find the value of c that makes it possible to solve Ax = b, and solve it: 


u+ yv+2w= 2 
2u + 3v -—- w= 5 
3u + 4V 4+ we= ee. 


. Under what conditions on b, and b» (if any) does Ax = b have a solution? 


A= 


12 0 3 pele! 
DA. 2 IB! 


Find two vectors in the nullspace of A, and the complete solution to Ax = b. 


. (a) Find the special solutions to Ux = 0. Reduce U to R and repeat: 


123.4) / 0 
U2 00 ONO AO 
v0 0a 0 

X4 


(b) If the right-hand side is changed from (0,0,0) to (a,b,0), what are all solutions? 
Find a 2 by 3 system Ax = b whose complete solution is 


1 
x= 12|+wI{3 
0 1 


11. 


12. 


13. 


14. 


15. 


16. 


17. 


18. 
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Find a 3 by 3 system with these solutions exactly when b; + bp = b3. 


Write a 2 by 2 system Ax = b with many solutions x, but no solution x,. (Therefore 
the system has no solution.) Which b’s allow an x)? 


Which of these rules give a correct definition of the rank of A? 


(a) The number of nonzero rows in R. 
(b) The number of columns minus the total number of rows. 
(c) The number of columns minus the number of free columns. 


(d) The number of Is in R. 
Find the reduced row echelon forms R and the rank of these matrices: 


(a) The 3 by 4 matrix of all 1s. 
(b) The 4 by 4 matrix with aj; = (—1)". 
(c) The 3 by 4 matrix with aj; = (—1)/. 


Find R for each of these (block) matrices, and the special solutions: 


_|A A 
lA O 


00 0 
A=1|0 0 3 B=|A Al C 
246 


If the r pivot variables come first, the reduced R must look like 


IF 
0 O 


R= Tisrbyr 
7 Fisrbyn—-r 


What is the nullspace matrix N containing the special solutions? 


Suppose all r pivot variables come /ast. Describe the four blocks in the m by n 
reduced echelon form (the block B should be r by r): 


A B 
C D 


R= 


What is the nullspace matrix N of special solutions? What is its shape? 


(Silly problem) Describe all 2 by 3 matrices A; and Az with row echelon forms R, 
and R>, such that R; + R> is the row echelon form of A; + A». Is it true that R; = A, 
and R> = A> in this case? 


If A has r pivot columns, then A! has r pivot columns. Give a 3 by 3 example for 
which the column numbers are different for A and A!. 


98 


19. 


20. 


21. 


22. 


23. 


24. 


25. 


26. 


27. 


28. 
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What are the special solutions to Rx = 0 and R'y = 0 for these R? 


LOY 2. 3 0 1 2 
R= |]0 145 R= |0 0 0 
00 0 0 0 0 0 


IfA has rank r, then it has an r by r submatrix S that is invertible. Find that submatrix 
S from the pivot rows and pivot columns of each A: 


{) 22 aa|i 2 3 
124 12 4 6 


Explain why the pivot rows and pivot columns of A (not R) always give an r by r 
invertible submatrix of A. 


A= 


0 1 0 
A=]0 0 0 
00 1 


Find the ranks of AB and AM (rank | matrix times rank | matrix): 
1 2 2 1 4 1 b 
2 i i ; 1.5 : - } 2 


Multiplying the rank 1 matrices A = uv! and B = wz! gives uz! times the number 
. AB has rank 1 unless =, 


A= 


Every column of AB is a combination of the columns of A. Then the dimensions of 
the column spaces give rank(AB) < rank(A). 


Problem: Prove also that rank(AB) < rank(B). 
(Important) Suppose A and B are n by n matrices, and AB = J. Prove from rank(AB) < 


rank(A) that the rank of A is n. So A is invertible and B must be its two-sided inverse. 
Therefore BA = I (which is not so obvious!). 


If A is 2 by 3 and C is 3 by 2, show from its rank that CA 4 J. Give an example in 
which AC = I. For m < n, a right inverse is not a left inverse. 


Suppose A and B have the same reduced-row echelon form R. Explain how to change 
A to B by elementary row operations. So B equals an matrix times A. 


Every m by n matrix of rank r reduces to (m by r) times (r by n): 
A = (pivot columns of A)(first r rows of R) = (COL)(ROW). 


Write the 3 by 4 matrix A at the start of this section as the product of the 3 by 2 
matrix from the pivot columns and the 2 by 4 matrix from R: 


23-2 
A=|2 697 
=. =3.34 


29. 


30. 


31. 


32. 


33. 


34. 


35. 
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Suppose A is an m by n matrix of rank r. Its reduced echelon form is R. Describe 
exactly the reduced row echelon form of R' (not A‘). 


(Recommended) Execute the six steps following equation (6) to find the column 
space and nullspace of A and the solution to Ax = b: 


246 4 by 4 
A=|2 57 6 b= Doi = 13 
D2. BoB b3 5 


For every c, find R and the special solutions to Ax = 0: 


l-c 2 


1 t-o9 
A=|12244 gn 0S Il" ee ; 
eee ‘ 


What is the nullspace matrix N (of special solutions) for A, B, C? 


I 


ae and eS lit Al: 


a=[r i] and a-| 


Problems 33-36 are about the solution of Ax = b. Follow the steps in the text to 
Xp» and x,. Reduce the augmented matrix [A )}. 


Find the complete solutions of 


x+3y+3z=1 1 34 3) |* 1 

2x+6y+92=5 and 26 A 28)? |= 13 
& 

=4 = 39-- 3k =) 002 4 ; 1 


Under what condition on b;, b2, b3 is the following system solvable? Include b as a 
fourth column in [A b]. Find all solutions when that condition holds: 
x+2y—2z=b), 
2x + 5y —4z = bo 
Ax + 9y — 8z = b3. 


What conditions on b;, b2, b3, b4 make each system solvable? Solve for x: 


12 a 123 bi 
2 4} [x] foo 24 6] \'| |» 
2 5| |x| |ba >a eames AG ical MP 
39 ba opal by 
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36. 


37. 
38. 


39. 


40. 


41. 


42. 


43. 


44. 
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Which vectors (b;,b2,b3) are in the column space of A? Which combinations of the 
rows of A give zero? 


(a) A= (b) A= 


oN Fe 
NY Ww 
nN WD 
NO —_- — 
RN 
ao fk eR 


Why can’t a | by 3 system have x, = (2,4,0) and x, = any multiple of (1,1, 1)? 


(a) If Ax = b has two solutions x; and x2, find two solutions to Ax = 0. 
(b) Then find another solution to Ax = b. 


Explain why all these statements are false: 


(a) The complete solution is any linear combination of x, and x;. 
(b) A system Ax = b has at most one particular solution. 


(c) The solution x, with all free variables zero is the shortest solution (minimum 
length ||x||). (Find a 2 by 2 counterexample.) 


(d) If A is invertible there is no solution x, in the nullspace. 
Suppose column 5 of U has no pivot. Then x5 is a variable. The zero vector 


(is) (is not) the only solution to Ax = 0. If Ax = b has a solution, then it has 
solutions. 


If you know x, (free variables = 0) and all special solutions for Ax = b, find x, and 
all special solutions for these systems: 
=|, 
M, = 
b 


If Ax = b has infinitely many solutions, why is it impossible for Ax = B (new right- 
hand side) to have only one solution? Could Ax = B have no solution? 


A 
A 


Ax = 2b 4 Al = 


Choose the number q so that (if possible) the ranks are (a) 1, (b) 2, (c) 3: 


G: @& 2 ee 
A=;-3 —2 -1l and B= | 5 | 
: © 4 q24q 


Give examples of matrices A for which the number of solutions to Ax = b is 


(a) 0 or 1, depending on b. 
(b) 0, regardless of b. 
(c) 0 or co, depending on b. 
(d) 1, regardless of b. 


45. 


46. 


47. 


48. 


49. 


50. 
51. 
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Write all known relations between r and m and n if Ax = b has 


(a) no solution for some b. 

(b) infinitely many solutions for every b. 

(c) exactly one solution for some b, no solution for other b. 
(d) exactly one solution for every b. 


Apply Gauss-Jordan elimination (right-hand side becomes extra column) to Ux = 0 
and Ux = c. Reach Rx = 0 and Rx = d: 


b -foae om be 


Solve Rx = 0 to find x, (its free variable is x. = 1). Solve Rx = d to find x, (its free 
variable is x7 = 0). 


1235 
00 4 8| 


Apply elimination with the extra column to reach Rx = 0 and Rx = d: 


3060 30 6 9 
lu 0) =|0 020 dita lu cl=|0 0 2 4 
0000 0005 


Solve Rx = 0 (free variable = 1). What are the solutions to Rx = d? 
Reduce to Ux = c (Gaussian elimination) and then Rx = d: 


“1 
x2 


3 
of |= [5] =e 
9 4 


Find a particular solution x, and all nullspace solutions x;. 

Find A and B with the given property or explain why you can’t. 
(a) The only solution to Ax = B isx= [9] 

(b) The only solution to Bx = [9] is x = B , 

The complete solution to Ax = [4] isx= [4] +c [9]. Find A. 
The nullspace of a 3 by 4 matrix A is the line through (2,3, 1,0). 


(a) What is the rank of A and the complete solution to Ax = 0? 


(b) What is the exact row reduced echelon form R of A? 
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52. Reduce these matrices A and B to their ordinary echelon forms U: 


122 46 (a a?) 
(a) A=]1 23 69 (b) B=|0 4 4 
001 2 3 0 8 8 


Find a special solution for each free variable and describe every solution to Ax = 0 
and Bx = 0. Reduce the echelon forms U to R, and draw a box around the identity 
matrix in the pivot rows and pivot columns. 


53. True or False? (Give reason if true, or counterexample to show it is false.) 


(a) A square matrix has no free variables. 
(b) An invertible matrix has no free variables. 
(c) An m by n matrix has no more than n pivot variables. 


(d) An m by n matrix has no more than m pivot variables. 
54. Is there a 3 by 3 matrix with no zero entries for which U = R= /? 


55. Put as many Is as possible in a 4 by 7 echelon matrix U and in a reduced form R 
whose pivot columns are 2, 4, 5. 


56. Suppose column 4 of a 3 by 5 matrix is all Os. Then x4 is certainly a variable. 
The special solution for this variable is the vector x = 


57. Suppose the first and last columns of a 3 by 5 matrix are the same (nonzero). Then 
is a free variable. Find the special solution for this variable. 


58. The equation x — 3y—z = 0 determines a plane in R*. What is the matrix A in this 
equation? Which are the free variables? The special solutions are (3,1,0) and___. 
The parallel plane x — 3y — z = 12 contains the particular point (12,0,0). All points 
on this plane have the following form (fill in the first components): 


Xx 
y| = |O| +y] 1] +z ]0 
Zz 0 0 1 


59. Suppose column | + column 3 + column 5 = 0 in a 4 by 5 matrix with four pivots. 
Which column is sure to have no pivot (and which variable is free)? What is the 
special solution? What is the nullspace? 


Problems 60-66 ask for matrices (if possible) with specific properties. 


60. Construct a matrix whose nullspace consists of all combinations of (2,2,1,0) and 
(3,1,0,1). 


61. Construct a matrix whose nullspace consists of all multiples of (4,3,2,1). 
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62. Construct a matrix whose column space contains (1,1,5) and (0,3.1) and whose 
nullspace contains (1, 1,2). 


63. Construct a matrix whose column space contains (1,1,0) and (0,1,1) and whose 
nullspace contains (1,0,1) and (0,0, 1). 


64. Construct a matrix whose column space contains (1,1, 1) and whose nullspace is the 
line of multiples of (1,1,1,1). 


65. Construct a 2 by 2 matrix whose nullspace equals its column space. 
66. Why does no 3 by 3 matrix have a nullspace that equals its column space? 


67. The reduced form R of a 3 by 3 matrix with randomly chosen entries is almost sure 
to be . What R is virtually certain if the random A is 4 by 3? 


68. Show by example that these three statements are generally false: 


(a) A and A! have the same nullspace. 
(b) A and A! have the same free variables. 
(c) If R is the reduced form rref(A) then R! is rref(A‘). 


69. If the special solutions to Rx = 0 are in the columns of these NV, go backward to find 
the nonzero rows of the reduced matrices R: 


23 0 
N=1]1 0 and N= |0 and N= (empty 3 by 1). 
0 1 1 


70. Explain why A and —A always have the same reduced echelon form R. 


2.3 Linear Independence, Basis, and Dimension 


By themselves, the numbers m and n give an incomplete picture of the true size of a 
linear system. The matrix in our example had three rows and four columns, but the third 
row was only a combination of the first two. After elimination it became a zero row, It 
had no effect on the homogeneous problem Ax = 0. The four columns also failed to be 
independent, and the column space degenerated into a two-dimensional plane. 

The important number that is beginning to emerge (the true size) is the rank r. The 
rank was introduced as the number of pivots in the elimination process. Equivalently, 
the final matrix U has r nonzero rows. This definition could be given to a computer. But 
it would be wrong to leave it there because the rank has a simple and intuitive meaning: 
The rank counts the number of genuinely independent rows in the matrix A. We want 
definitions that are mathematical rather than computational. 

The goal of this section is to explain and use four ideas: 
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1. Linear independence or dependence. 
2. Spanning a subspace. 

3. Basis for a subspace (a set of vectors). 
4. Dimension of a subspace (a number). 


The first step is to define linear independence. Given a set of vectors v,,...,Vx%, We 
look at their combinations c,vj + cov2 +---+c,vz. The trivial combination, with all 
weights c; = 0, obviously produces the zero vector: Ovy +---+ Ov, = 0. The question is 
whether this is the only way to produce zero. If so, the vectors are independent. 

If any other combination of the vectors gives zero, they are dependent. 


2E Suppose cjvj +:--+cgvg =0 only happens when c; =--- = cy, = 0. Then 
the vectors v1,...,v% are linearly independent. If any c’s are nonzero, the v’s 
are linearly dependent. One vector is a combination of the others. 


Linear dependence is easy to visualize in three-dimensional space, when all vectors 
go out from the origin. Two vectors are dependent if they lie on the same line. Three 
vectors are dependent if they lie in the same plane. A random choice of three vectors, 
without any special accident, should produce linear independence (not in a plane). Four 
vectors are always linearly dependent in R?. 


Example 1. If v; = zero vector, then the set is linearly dependent. We may choose 
c, = 3 and all other c; = 0; this is a nontrivial combination that produces zero. 


Example 2. The columns of the matrix 


are linearly dependent, since the second column is three times the first. The combination 
of columns with weights —3, 1, 0, 0 gives a column of zeros. 

The rows are also linearly dependent; row 3 is two times row 2 minus five times row 
1. (This is the same as the combination of b1, b2, b3, that had to vanish on the right-hand 
side in order for Ax = b to be consistent. Unless b3 — 2b2 + 5b; = 0, the third equation 
would not become 0 = 0.) 


Example 3. The columns of this triangular matrix are linearly independent: 


No zeros on the diagonal A= 


ow 
orn 
NOES) 
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Look for a combination of the columns that makes zero: 


3 4 2 0 
Solve Ac = 0 cy |}O}| +c2 | 1] +c3 |5] = |0 
0 0 2 0 


We have to show that cj, c2, c3 are all forced to be zero. The last equation gives 
c3 = 0. Then the next equation gives cz = 0, and substituting into the first equation forces 
c, = 0. The only combination to produce the zero vector is the trivial combination. The 
nullspace of A contains only the zero vector c, = c2 = c3 = 0. 


The columns of A are independent exactly when N(A) = {zero vector}. 


A similar reasoning applies to the rows of A, which are also independent. Suppose 
c1(3,4,2) + c2(0, 1,5) +¢3(0,0,2) = (0,0,0). 


From the first components we find 3c; = 0 or c; = 0. Then the second components give 
c2 = 0, and finally c3 = 0. 

The nonzero rows of any echelon matrix U must be independent. Furthermore, if we 
pick out the columns that contain the pivots, they also are linearly independent. In our 
earlier example, with 


Two independent rows 
Two independent columns 


the pivot columns | and 3 are independent. No set of three columns is independent, and 
certainly not all four. It is true that columns 1 and 4 are also independent, but if that 
last | were changed to 0 they would be dependent. /t is the columns with pivots that are 
guaranteed to be independent. The general rule is this: 


2F The r nonzero rows of an echelon matrix U and a reduced matrix R are 
linearly independent. So are the r columns that contain pivots. 


Example 4. The columns of the 1 by n identity matrix are independent: 


10 .- 0 
O01 .- 0 
l= 
0 
0001 
These columns e),...,é) represent unit vectors in the coordinate directions; in R‘, 
1 0 0) 0 
0 ; 1 ; 0 0 
ey, = 5 — 5 — 5 — 

a> 1G 2 NG a ie cama 
0 0 0 1 
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Most sets of four vectors in R* are independent. Those e’s might be the safest. 


To check any set of vectors v1,...,V, for independence, put them in the columns of A. 
Then solve the system Ac = 0; the vectors are dependent if there is a solution other than 
c = 0. With no free variables (rank n), there is no nullspace except c = 0; the vectors are 
independent. If the rank is less than n, at least one free variable can be nonzero and the 
columns are dependent. 

One case has special importance. Let the n vectors have m components, so that A is an 
m by n matrix. Suppose now that n > m. There are too many columns to be independents 
There cannot be n pivots, since there are not enough rows to hold them. The rank will 
be less than n. Every system Ac = 0 with more unknowns than equations has solutions 


CSO: 


2G A set of n vectors in R” must be linearly dependent if n > m. 


The reader will recognize this as a disguised form of 2C: Every m by n system Ax = 0 
has nonzero solutions if n > m. 


Example 5. These three columns in R” cannot be independent: 


fi 25 1 
A= ; 
i 4 
To find the combination of the columns producing zero we solve Ac = 0: 
ie a | 
A-U= : 
0 1 1 
If we give the value 1 to the free variable c3, then back-substitution in Uc = 0 gives 
cz = —1, c; = 1. With these three weights, the first column minus the second plus the 


third equals zero: Dependence. 


Spanning a Subspace 


Now we define what it means for a set of vectors to span a space. The column space of 
A 1s spanned by the columns. Their combinations produce the whole space: 


2H_ Ifa vector space V consists of all linear combinations of w1,...,we, then 
these vectors span the space. Every vector v in V is some combination of the 
w’s: 


Every v comes from w’s v=ciwi+--:-+cewe for some coefficients c;. 


It is permitted that a different combination of w’s could give the same vector v. The 
c’s need not be unique, because the spanning set might be excessively large—it could 
include the zero vector, or even all vectors. 
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Example 6. The vectors w; = (1,0,0), w2 = (0,1,0), and w3 = (—2,0,0) span a plane 
(the x-y plane) in R*. The first two vectors also span this plane, whereas w; and w3 span 
only a line. 


Example 7. The column space of A is exactly the space that is spanned by its columns. 
The row space is spanned by the rows. The definition is made to order. Multiplying A 
by any x gives a combination of the columns; it is a vector Ax in the column space. 

The coordinate vectors e;,...,@, coming from the identity matrix span R”. Every 
vector b = (bj,...,b,) is a combination of those columns. In this example the weights 
are the components b; themselves: b = bye; +--+ bnen. But the columns of other 
matrices also span R"! 


Basis for a Vector Space 


To decide if b is a combination of the columns, we try to solve Ax = b. To decide if 
the columns are independent, we solve Ax = 0. Spanning involves the column space, 
and independence involves the nullspace. The coordinate vectors e1,...,é@, span R” 
and they are linearly independent. Roughly speaking, no vectors in that set are wasted. 
This leads to the crucial idea of a basis. 


2| A basis for V is a sequence of vectors having two properties at once: 


1. The vectors are linearly independent (not too many vectors). 


2. They span the space V (not too few vectors). 


This combination of properties is absolutely fundamental to linear algebra. It means 
that every vector in the space is a combination of the basis vectors, because they span. 
It also means that the combination is unique: If v = a,;vy +---+ agvg and also v = 
biv; +--+: + bgvg, then subtraction gives 0 = )\(a; — b;)v;. Now independence plays its 
part; every coefficient a; — b; must be zero. Therefore a; = b;. There is one and only 
one way to write v as a combination of the basis vectors. 

We had better say at once that the coordinate vectors e1,...,é, are not the only basis 
for R”. Some things in linear algebra are unique, but not this. A vector space has 
infinitely many different bases. Whenever a square matrix is invertible, its columns are 
independent—and they are a basis for R”. The two columns of this nonsingular matrix 
are a basis for R7: 


Every two-dimensional vector is a combination of those (independent!) columns. 


Example 8. The x-y plane in Figure 2.4 is just R*. The vector v; by itself is linearly 
independent, but it fails to span R?. The three vectors v1, v2, v3 certainly span R’, but 
are not independent. Any two of these vectors, say v; and v2, have both properties—they 
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span, and they are independent. So they form a basis. Notice again that a vector space 
does not have a unique basis. 


U3 


U2 


UI 


Figure 2.4: A spanning set v1, v2, v3. Bases v1, v2 and vj, v3 and vg, v3. 


Example 9. These four columns span the column space of U, but they are not indepen- 


dent: 
13 3 2 


Echelon matrix U=10 03 1 
000 0 


There are many possibilities for a basis, but we propose a specific choice: The columns 
that contain pivots (in this case the first and third, which correspond to the basic vari- 
ables) are a basis for the column space. These columns are independent, and it is easy 
to see that they span the space. In fact, the column space of U is just the x-y plane 
within R°. C(U) is not the same as the column space C(A) before elimination—but the 
number of independent columns didn’t change. 


To summarize: The columns of any matrix span its column space. If they are indepen- 
dent, they are a basis for the column space—whether the matrix is square or rectangular. 
If we are asking the columns to be a basis for the whole space R”, then the matrix must 
be square and invertible. 


Dimension of a Vector Space 


A space has infinitely many different bases, but there is something common to all of 
these choices. The number of basis vectors is a property of the space itself: 


2J Any two bases for a vector space V contain the same number of vec- 
tors. This number, which is shared by all bases and expresses the number of 
“degrees of freedom” of the space, is the dimension of V. 


We have to prove this fact: All possible bases contain the same number of vectors. 
The x-y plane in Figure 2.4 has two vectors in every basis; its dimension is 2. In three 
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dimensions we need three vectors, along the x-y-z axes or in three other (linearly in- 
dependent!) directions. The dimension of the space R" is n. The column space of U 
in Example 9 had dimension 2; it was a “two-dimensional subspace of R°.” The zero 
matrix is rather exceptional, because its column space contains only the zero vector. By 
convention, the empty set is a basis for that space, and its dimension is zero. 

Here is our first big theorem in linear algebra: 


2K Ifvj,...,Vm and w1,...,w, are both bases for the same vector space, then 
m =n. The number of vectors is the same. 


Proof. Suppose there are more w’s than v’s (n > m). We will arrive at a contradiction. 
Since the v’s form a basis, they must span the space. Every w; can be written as a 
combination of the v’s: If wy = a4,vj +-+:+@m1Vm, this is the first column of a matrix 
multiplication VA: 


w= [wi W2 °°: wn] = [v1 ves vm! : =VA. 
aml 


We don’t know each a;;, but we know the shape of A (it is m by n). The second vector 
W2 is also a combination of the v’s. The coefficients in that combination fill the second 
column of A. The key is that A has a row for every v and a column for every w. A is a 
short, wide matrix, since n > m. There is a nonzero solution to Ax = 0. Then VAx = 0 
which is Wx = 0. A combination of the w’s gives zero! The w’s could not be a basis—so 
we cannot have n > m. 

If m > n we exchange the v’s and w’s and repeat the same steps. The only way to 
avoid a contradiction is to have m = n. This completes the proof that m = n. To repeat: 
The dimension of a space is the number of vectors in every basis. 


This proof was used earlier to show that every set of m+ 1 vectors in R” must be de- 
pendent. The v’s and w’s need not be column vectors—the proof was all about the matrix 
A of coefficients. In fact we can see this general result: In a subspace of dimension k, 
no set of more than k vectors can be independent, and no set of more than k vectors can. 
span the space. 

There are other “dual” theorems, of which we mention only one. We can start with a 
set of vectors that is too small or too big, and end up with a basis: 


2L Any linearly independent set in V can be extended to a basis, by adding 
more vectors if necessary. 


Any spanning set in V can be reduced to a basis, by discarding vectors if 
necessary. 


The point is that a basis is a maximal independent set. It cannot be made larger without 
losing independence. A basis is also a minimal spanning set. It cannot be made smaller 
and still span the space. 
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You must notice that the word “dimensional” is used in two different ways. We speak 
about a four-dimensional vector, meaning a vector in R*. Now we have defined a four- 
dimensional subspace; an example is the set of vectors in R° whose first and last com- 
ponents are zero. The members of this four-dimensional subspace are six-dimensional 
vectors like (0,5,1,3,4,0). 

One final note about the language of linear algebra. We never use the terms “basis of a 
matrix” or “rank of a space” or “dimension of a basis.” These phrases have no meaning. 
It is the dimension of the column space that equals the rank of the matrix, as we prove 
in the coming section. 


Problem Set 2.3 


Problems 1-10 are about linear independence and linear dependence. 


1. Show that v;, v2, v3 are independent but v1, v2, v3, v4 are dependent: 


1 1 1 2 
y= |0 v= |1 Va el vw4= {3 
0 0 1 4 


Solve cjvj +---+cav4 = 0 or Ac = 0. The v’s go in the columns of A. 


2. Find the largest possible number of independent vectors among 


1 1 1 0 0 0 
—1 0 0 ; 1 F 1 : 0 
Yi = w= y2= — = = . 
SS eGgh) Oe ep a, rea) ore gl! See 
0 0 —1 0 —1 —1 
This number is the of the space spanned by the v’s. 


3. Prove that if a= 0, d = 0, or f = 0 (3 cases), the columns of U are dependent: 


Gob: 
U=10 de 
CoD. Ff 


4. Ifa,d, f in Problem 3 are all nonzero, show that the only solution to Ux = 0 is x = 0. 
Then U has independent columns. 

5. Decide the dependence or independence of 
(a) the vectors (1,3,2), (2,1,3), and (3.2, 1). 
(b) the vectors (1, —3,2), (2,1, -—3), and (—3,2,1). 


10. 


11. 


12. 


13. 
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Choose three independent columns of U. Then make two other choices. Do the same 
for A. You have found bases for which spaces? 


ieee 2341 
ee eae Be ag ce PO Oe to 
0009 0009 

0000 4682 


If w1, w2, w3 are independent vectors, show that the differences vj = w2 — w3, v2 = 
W, — W3, and v3 = wy — W2 are dependent. Find a combination of the v’s that gives 
zero. 


If w1, w2, w3 are independent vectors, show that the sums vj = w2+W3, v2 =W1+W3, 
and v3 = w1 + W2 are independent. (Write c, vj + c2v2 + ¢3V3 = 0 in terms of the w’s. 
Find and solve equations for the c’s.) 


Suppose v1, v2, v3, v4 are vectors in R’. 


(a) These four vectors are dependent because ; 

(b) The two vectors v; and v2 will be dependent if , 

(c) The vectors v; and (0,0,0) are dependent because 

Find two independent vectors on the plane x + 2y — 3z —t = 0 in R‘. Then find three 
independent vectors. Why not four? This plane is the nullspace of what matrix? 
Problems 11-18 are about the space spanned by a set of vectors. Take all linear 
combinations of the vectors 

Describe the subspace of R? (is it a line or a plane or R*?) spanned by 

(a) the two vectors (1,1,—1) and (—1,—1,1). 

(b) the three vectors (0, 1,1) and (1,1,0) and (0,0,0). 

(c) the columns of a 3 by 5 echelon matrix with 2 pivots. 

(d) all vectors with positive components. 

The vector b is in the subspace spanned by the columns of A when there is a solution 


to . The vector c is in the row space of A when there is a solution to . True 
or false: If the zero vector is in the row space, the rows are dependent. 


Find the dimensions of (a) the column space of A, (b) the column space of U, (c) the 
row space of A, (d) the row space of U. Which two of the spaces are the same? 


1 1 1 1 0 
A=/1 3 1 and U=j|0 2 1 
See 0 0 0 
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14. 


15. 


16. 


17. 


18. 


19. 


20. 


21. 


22. 
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Choose x = (x1,%2,%3,%4) in R*. It has 24 rearrangements like (x2,X1,X3,x4) and 
(x4,X3,X1,X2). Those 24 vectors, including x itself, span a subspace S. Find specific 
vectors x so that the dimension of S is: (a) 0, (b) 1, (c) 3, (d) 4. 


v+w and v—w are combinations of v and w. Write v and w as combinations of v+w 
and v —w. The two pairs of vectors the same space. When are they a basis for 
the same space? 


Decide whether or not the following vectors are linearly independent, by solving 
C{V] + C2V2 + €3V3 + cav4 = 0: 


Y= 


S 

L 

| 
eS aS 


Decide also if they span R’, by trying to solve cjvj +---+c4v4 = (0,0,0, 1). 


Suppose the vectors to be tested for independence are placed into the rows instead 
of the columns of A, How does the elimination process from A to U decide for or 
against independence? 


To decide whether b is in the sub space spanned by wj,...,w,, let the vectors w be 
the columns of A and try to solve Ax = b. What is the result for 


(a) w; =(1,1,0), wo = (2,2, 1), w3 = (0,0,2), b = (3,4,5)? 
(b) wy = (1,2,0), w2 = (2,5,0), w3 = (0,0,2), w4 = (0,0,0), and any b? 


Problems 19-37 are about the requirements for a basis. 


If vj,...,V¥, are linearly independent, the space they span has dimension . These 
vectors are a for that space. If the vectors are the columns of an m by n matrix, 
then m is than n. 


Find a basis for each of these subspaces of R*: 


(a) All vectors whose components are equal. 
(b) All vectors whose components add to zero. 
(c) All vectors that are perpendicular to (1,1,0,0) and (1,0,1,1). 


(d) The column space (in R’) and nullspace (in R°) of U = asco 


Find three different bases for the column space of U above. Then find two different 
bases for the row space of U. 


Suppose v1,V2,...,¥6 are Six vectors in R’. 


(a) Those vectors (do)(do not)(might not) span R‘. 


23. 


24. 


23s 


26. 


27. 


28. 


29. 
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(b) Those vectors (are)(are not)(might be) linearly independent. 
(c) Any four of those vectors (are)(are not)(might be) a basis for R‘. 


(d) If those vectors are the columns of A, then Ax = b (has) (does not have) (might 
not have) a solution. 


The columns of A are n vectors from R”. If they are linearly independent, what is 
the rank of A? If they span R”, what is the rank? If they are a basis for R”, what 
then? 


Find a basis for the plane x — 2y +3z = 0 in R®. Then find a basis for the intersection 
of that plane with the xy-plane. Then find a basis for all vectors perpendicular to the 
plane. 


Suppose the columns of a 5 by 5 matrix A are a basis for R°. 


(a) The equation Ax = 0 has only the solution x = 0 because : 
(b) If b is in R° then Ax = b is solvable because ; 


Conclusion: A is invertible. Its rank is 5. 
Suppose S is a five-dimensional subspace of R°. True or false? 


(a) Every basis for S can be extended to a basis for R° by adding one more vector. 


(b) Every basis for R° can be reduced to a basis for S by removing one vector. 


U comes from A by subtracting row | from row 3: 


L. -3 2 1B 2D 
A=|0 1 1 and U=j|0 1 1 
Loe 0 0 0 


Find bases for the two column spaces. Find bases for the two row spaces. Find bases 
for the two nullspace. 


True or false (give a good reason)? 


(a) If the columns of a matrix are dependent, so are the rows. 
(b) The column space of a 2 by 2 matrix is the same as its row space. 
(c) The column space of a 2 by 2 matrix has the same dimension as its row space. 


(d) The columns of a matrix are a basis for the column space. 


For which numbers c and d do these matrices have rank 2? 
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30. By locating the pivots, find a basis for the column space of 


Express each column that is not in the basis as a combination of the basic columns, 
Find also a matrix A with this echelon form U, but a different column space. 


31. Find a counterexample to the following statement: If vj, v2, v3, v4 is a basis for the 
vector space R+, and if W is a subspace, then some subset of the v’s is a basis for W. 


32. Find the dimensions of these vector spaces: 


(a) The space of all vectors in R* whose components add to zero. 
(b) The nullspace of the 4 by 4 identity matrix. 
(c) The space of all 4 by 4 matrices. 


33. Suppose V is known to have dimension k. Prove that 


(a) any k independent vectors in V form a basis; 


(b) any k vectors that span V form a basis. 


In other words, if the number of vectors is known to be correct, either of the two 
properties of a basis implies the other. 


34. Prove that if V and W are three-dimensional subspaces of R°, then V and W must 
have a nonzero vector in common. Hint: Start with bases for the two subspaces, 
making six vectors in all. 


35. True or false? 


(a) If the columns of A are linearly independent, then Ax = b has exactly one solution 
for every b. 


(b) A 5 by 7 matrix never has linearly independent columns, 


36. If A is a 64 by 17 matrix of rank 11, how many independent vectors satisfy Ax = 0? 
How many independent vectors satisfy Aly = 0? 


37. Find a basis for each of these subspaces of 3 by 3 matrices: 


(a) All diagonal matrices. 
(b) All symmetric matrices (A! = A). 


(c) All skew-symmetric matrices (A! = —A). 


Problems 38—42 are about spaces in which the “‘vectors” are functions. 


38. 


Re 


40. 


41. 


42. 


43. 


44. 


45. 
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(a) Find all functions that satisfy $ ay — (), 
(b) Choose a particular function that satisfies 4 ix = 3 


(c) Find all functions that satisfy ay = 


The cosine space F3 contains all combinations y(x) = Acosx + Bcos2x + Ccos 3x. 
Find a basis for the subspace that has y(0) = 0. 


Find a basis for the space of functions that satisfy 


d 
(a) = —2y=0. 
ty y 
b) —--=0 
Or x 


Suppose yj (x), y2(x), y3(x) are three different functions of x. The vector space they 
span could have dimension 1, 2, or 3. Give an example of yj, y2, y3 to show each 
possibility. 

Find a basis for the space of polynomials p(x) of degree < 3. Find a basis for the 
subspace with p(1) = 0. 


Write the 3 by 3 identity matrix as a combination of the other five permutation matri- 
ces! Then show that those five matrices are linearly independent. (Assume a combi 
nation gives zero, and check entries to prove each term is zero.) The five permuta- 
tions are a basis for the subspace of 3 by 3 matrices with row and column sums all 
equal. 


Review: Which of the following are bases for R>? 
(a) (1,2,0) and (0,1,—1). 

(b) (1,1,-1), (2,3,4), (4,1,-1), (0,1,-1). 

(c) (1,2,2), (—1,2, 1), (0,8,0). 

(d) (1,2,2), (—1,2,1), (0,8,6). 


Review: Suppose A is 5 by 4 with rank 4. Show that Ax = b has no solution when the 
5 by 5 matrix [A )] is invertible. Show that Ax = b is solvable when [A 5] is singular. 


2.4 The Four Fundamental Subspaces 


The previous section dealt with definitions rather than constructions. We know what a 
basis is, but not how to find one. Now, starting from an explicit description of a subspace, 
we would like to compute an explicit basis. 


Subspaces can be described in two ways. First, we may be given a set of vectors that 


span the space. (Example: The columns span the column space.) Second, we may be 
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told which conditions the vectors in the space must satisfy. (Example: The nullspace 
consists of all vectors that satisfy Ax = 0.) 

The first description may include useless vectors (dependent columns). The second 
description may include repeated conditions (dependent rows). We can’t write a basis 
by inspection, and a systematic procedure is necessary. 

The reader can guess what that procedure will be. When elimination on A produces 
an echelon matrix U or a reduced R, we will find a basis for each of the subspaces 
associated with A. Then we have to look at the extreme case of full rank: 


When the rank is as large as possible, r=n orr=morr=m=n, the matrix has a 
left-inverse B or a right-inverse C or a two-sided A~!. 


To organize the whole discussion, we take each of the four subspaces in turn. Two of 
them are familiar and two are new. 


1. The column space of A is denoted by C(A). Its dimension is the rank r. 
2. The nullspace of A is denoted by N(A). Its dimension is n — r. 


3. The row space of A is the column space of A’. It is C(A'), and it is spanned by the 
rows of A. Its dimension is also r. 


4. The left nullspace of A is the nullspace of A‘. It contains all vectors y such that 
Aly =0, and it is written N(A‘). Its dimension is 


The point about the last two subspaces is that they come from A‘. If A is an m by n 
matrix, you can see which “host” spaces contain the four subspaces by looking at the 
number of components: 


The nullspace N(A) and row space C(A!) are subspaces of R". 
The left nullspace N(A‘) and column space C'(A) are subspaces of R”. 


The rows have n components and the columns have m. For a simple matrix like 
100 
0 0 Of’ 


the column space is the line through [}]. The row space is the line through [1 0 0’. It 
is in R°. The nullspace is a plane in R? and the left nullspace is a line in R’: 


0 0 
; T ; 0 
N(A) contains [1] and |/0/, N(A’) contains ae 
0 1 
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Note that all vectors are column vectors. Even the rows are transposed, and the row 
space of A is the column space of A‘, Our problem will be to connect the four spaces for 
U (after elimination) to the four spaces for A: 


eu 1332 1 3 32 
an UZ 6.00% 3\\ sanehonr A239. “6 O95 
ec ta 0000 Lf 2363-4 


For novelty, we take the four subspaces in a more interesting order. 


3. The row space of A For an echelon matrix like U, the row space is clear. It contains 
all combinations of the rows, as every row space does—but here the third row contributes 
nothing. The first two rows are a basis for the row space. A similar rule applies to every 
echelon matrix U or R, with r pivots and r nonzero rows: The nonzero rows are a basis, 
and the row space has dimension r. That makes it easy to deal with the original matrix 
A. 


2M The row space of A has the same dimension r as the row space of U, 
and it has the same bases, because the row spaces of A and U (and R) are the 
same. 


The reason is that each elementary operation leaves the row space unchanged. The rows 
in U are combinations of the original rows in A. Therefore the row space of U contains 
nothing new. At the same time, because every step can be reversed, nothing is lost; the 
rows of A can be recovered from U. It is true that A and U have different rows, but the 
combinations of the rows are identical: same space! 

Note that we did not start with the m rows of A, which span the row space, and discard 
m—r of them to end up with a basis. According to 2L, we could have done so. But it 
might be hard to decide which rows to keep and which to discard, so it was easier just to 
take the nonzero rows of U. 


2. The nullspace of A Elimination simplifies a system of linear equations without 
changing the solutions. The system Ax = 0 is reduced to Ux = 0, and this process is 
reversible. The nullspace of A is the same as the nullspace of U and R. Only r of 
the equations Ax = 0 are independent. Choosing the n —r “special solutions” to Ax = 0 
provides a definite basis for the nullspace: 


2N_ The nullspace N(A) has dimension n — r. The “special solutions” are a 
basis—each free variable is given the value 1, while the other free variables 
are 0. Then Ax = 0 or Ux = 0 or Rx = 0 gives the pivot variables by back- 
substitution. 


This is exactly the way we have been solving Ux = 0. The basic example above has 
pivots in columns 1 and 3. Therefore its free variables are the second and fourth v and y. 
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The basis for the nullspace is 


=3 1 

; i v= l 1 y= 0 0 
Special solutions i= ; X= 

y = 0 0 = | | 

0 1 


Any combination cx; + c2x2 has c, as its vy component, and cz as its y component. The 
only way to have cx; +c2x2 = 0 is to have c; = cz = 0, so these vectors are independent. 
They also span the nullspace; the complete solution is vx; + yx2. Thus then —r=4—2 
vectors are a basis. 

The nullspace is also called the kernel of A, and its dimension n — r is the nullity. 


1. The column space of A The column space is sometimes called the range. This 
is consistent with the usual idea of the range, as the set of all possible values f(x); x is 
in the domain and f(x) is in the range. In our case the function is f(x) = Ax. Its domain 
consists of all x in R”; its range is all possible vectors Ax, which is the column space. 
(In an earlier edition of this book we called it R(A).) 

Our problem is to find bases for the column spaces of U and A. Those spaces are 
different (just look at the matrices!) but their dimensions are the same. 

The first and third columns of U are a basis for its column space. They are the 
columns with pivots. Every other column is a combination of those two. Furthermore, 
the same is true of the original A—even though its columns are different. The pivot 
columns of A are a basis for its column space. The second column is three times 
the first, just as in U. The fourth column equals (column 3) — (column 1). The same 
nullspace is telling us those dependencies. 

The reason is this: Ax = 0 exactly when Ux = 0. The two systems are equivalent and 
have the same solutions. The fourth column of U was also (column 3) — (column 1). 
Every linear dependence Ax = 0 among the columns of A is matched by a dependence 
Ux = 0 among the columns of U, with exactly the same coefficients. [fa set of columns 
of A is independent, then so are the corresponding columns of U, and vice versa. 

To find a basis for the column space C(A), we use what is already done for U. The 
r columns containing pivots are a basis for the column space of U. We will pick those 
same r columns in A: 


20 The dimension of the column space CA) equals the rank r, which also 
equals the dimension of the row space: The number of independent columns 
equals the number of independent rows. A basis for C(A) is formed by the r 
columns of A that correspond, in U, to the columns containing pivots. 


The row space and the column space have the same dimension r! This is one of 
the most important theorems in linear algebra. It is often abbreviated as “row rank = 
column rank.” It expresses a result that, for a random 10 by 12 matrix, is not at all 
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obvious. It also says something about square matrices: If the rows of a square matrix 
are linearly independent, then so are the columns (and vice versa). Again, that does not 
seem self-evident (at least, not to the author). 

To see once more that both the row and column spaces of U have dimension r, con- 
sider a typical situation with rank r = 3. The echelon matrix U certainly has three 
independent rows: 


dj * * * * * 
ps | Ook Oe 
10 00 0 O0fd; 


000000. 


We claim that U also has three independent columns, and no more, The columns have 
only three nonzero components. If we can show that the pivot columns—the first, fourth, 
and sixth—are linearly independent, they must be a basis (for the column space of U, 
not A!). Suppose a combination of these pivot columns produced zero: 


d, * * 0 
0 d k 0 
Cl +02 0 +03 “is = 0 
0 0 0 0 


Working upward in the usual way, c3 must be zero because the pivot d3 4 0, then cz must 
be zero because dy # 0, and finally c; = 0. This establishes independence and completes 
the proof. Since Ax = 0 if and only if Ux = 0, the first, fourth, and sixth columns of A— 
whatever the original matrix A was, which we do not even know in this example—are a 
basis for C(A). 

The row space and column space both became clear after elimination on A. Now 
comes the fourth fundamental subspace, which has been keeping quietly out of sight. 
Since the first three spaces were C(A), N(A), and C(A'), the fourth space must be 
N(A!), It is the nullspace of the transpose, or the left nullspace of A. Aly = 0 means 
y'A = 0, and the vector appears on the left-hand side of A. 


4. The left nullspace of A (= the nullspace of A‘) If A is an m by n matrix, then 
A! isn by m. Its nullspace is a subspace of R”; the vector y has m components. Written 
as y'A = 0, those components multiply the rows of A to produce the zero row: 


yTA = [y1 ies ¥m| | A |=[0 a 0). 


The dimension of this nullspace N (AT) is easy to find, For any matrix, the number 
of pivot variables plus the number of free variables must match the total number of 
columns. For A, that was r+ (n—r) =n. In other words, rank plus nullity equals n: 


dimension of C(A) + dimension of N(A) = number of columns. 
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This law applies equally to A', which has m columns. A? is just as good a matrix as A. 
But the dimension of its column space is also r, so 


r+ dimension (N(A‘)) =m. (1) 
2P The left nullspace N(A‘) has dimension m — r. 


The m—r solutions to y'A = 0 are hiding somewhere in elimination. The rows of A 
combine to produce the m—r zero rows of U. Start from PA = LU, or L~'PA = U. The 
last m—r rows of the invertible matrix L~'P must be a basis of y’s in the left nullspace— 
because they multiply A to give the zero rows in U. 

In our 3 by 4 example, the zero row was row 3 — 2(row 2) + S5(row 1). Therefore 
the components of y are 5, —2, 1. This is the same combination as in b3 — 2b2 + 5b, on 
the right-hand side, leading to 0 = 0 as the final equation. That vector y is a basis for 
the left nullspace, which has dimension m—r=3—2= 1. It is the last row of L~'P, 
and produces the zero row in U—and we can often see it without computing L~'. When 
desperate, it is always possible just to solve Aly = 0. 

I realize that so far in this book we have given no reason to care about N(A!). It is 
correct but not convincing if I write in italics that the left nullspace is also important. The 
next section does better by finding a physical meaning for y from Kirchhoff’s Current 
Law. 

Now we know the dimensions of the four spaces. We can summarize them in a table, 
and it even seems fair to advertise them as the 


Fundamental Theorem of Linear Algebra, Part I 
1. C(A) =column space of A; dimension r. 
2. N(A) = nullspace of A; dimension n — r. 
3. C(A!) = row space of A; dimension r. 
4 


. N(A‘) = left nullspace of A; dimension m — r. 


Example 1. A = 


Lb? 
has m =n = 2, and rank r= 1. 
3 6 
1. The column space contains all multiples of ah The second column is in the same 
direction and contributes nothing new. 
2. The nullspace contains all multiples of eal . This vector satisfies Ax = 0. 


3. The row space contains all multiples of eee I write it as a column vector, since 
strictly speaking it is in the column space of A!. 


4. The left nullspace contains all multiples of y = eae The rows of A with coeffi- 
cients —3 and 1 add to zero, so A'y = 0. 
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In this example all four subspaces are lines. That is an accident, coming from 7 = 1 and 
n—r=1andm—r=1. Figure 2.5 shows that two pairs of lines are perpendicular. That 
is no accident! 


column space 
C(A) 
multiples of (1,3) 


_  nullspace NV (AT) 
multiples of (38, —1) 


row space C(AT) | nullspace N(A) Pe : | 
multiples of (1,2) | multiples of (2, —1) — [3 6 


Figure 2.5: The four fundamental subspaces (lines) for the singular matrix A. 


If you change the last entry of A from 6 to 7, all the dimensions are different. The 
column space and row space have dimension r = 2. The nullspace and left nullspace 
contain only the vectors x = 0 and y = 0. The matrix is invertible. 


Existence of Inverses 


We know that if A has a left-inverse (BA = /) and a right-inverse (AC = J), then the two 
inverses are equal: B = B(AC)(BA)C = C. Now, from the rank of a matrix, it is easy to 
decide which matrices actually have these inverses. Roughly speaking, an inverse exists 
only when the rank is as large as possible. 

The rank always satisfies r < m and also r <n. An m by n matrix cannot have more 
than m independent rows or n independent columns. There is not space for more than m 
pivots, or more than n. We want to prove that when r = m there is a right-inverse, and 
Ax = b always has a solution. When r = n there is a left-inverse, and the solution (if it 
exists) 18 unique. 

Only a square matrix can have both r = m and r = n, and therefore only a square 
matrix can achieve both existence and uniqueness. Only a square matrix has a two-sided 
inverse. 


2Q EXISTENCE: Full row rank r = m. Ax = b has at least one solution x 
for every b if and only if the columns span R”. Then A has a right-inverse C 
such that AC = I, (m by m). This is possible only if m <n. 


UNIQUENESS: Full column rank r = n. Ax = b has at most one solution x 
for every b if and only if the columns are linearly independent. Then A has an 
n by m left-inverse B such that BA = J,,. This is possible only if m > n. 
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In the existence case, one possible solution is x = Cb, since then Ax = ACb = b. But 
there will be other solutions if there are other right-inverses. The number of solutions 
when the columns span R” is 1 or ©. 

In the uniqueness case, if there is a solution to Ax = J, it has to be x = BAx = Bb. But 
there may be no solution. The number of solutions is 0 or 1. 

There are simple formulas for the best left and right inverses, if they exist: 


One-sided inverses B=(A'A)'A? and C=Al(AA!) |, 


Certainly BA = I and AC = J. What is not so certain is that A'A and AA? are actually 
invertible. We show in Chapter 3 that ATA does have an inverse if the rank is n, and AAT 
has an inverse when the rank is m. Thus the formulas make sense exactly when the rank 
is as large as possible, and the one-sided inverses are found. 


Example 2. Consider a simple 2 by 3 matrix of rank 2: 


ea 4 0 0 . 
05 0 
Since r = m = 2, the theorem guarantees a right-inverse C: 
I 
5; O 
4 0 0| | 4 1 0 
AC = 0 il= ; 
05 ; 0 
C31 €32 


There are many right-inverses because the last row of C is completely arbitrary. This is 
a case of existence but not uniqueness. The matrix A has no left-inverse because the last 
column of BA is certain to be zero. The specific right-inverse C = A'(AA')~! chooses 
C31 and c32 to be zero: 


BOON Ta op i 0 
Best right-inverse A'(AA')"'= |0 5 4 = 10 5 =€. 
0 0 ze 0 0 


This is the pseudoinverse—a way of choosing the best C in Section 6.3. The transpose 
of A yields an example with infinitely many J/eft-inverses: 


4 0 
; 0 by; Ae taslee 
0 4 by Od) 
Now it is the last column of B that is completely arbitrary. The best left-inverse (also the 
pseudoinverse) has b;3 = b23 = 0. This is a “uniqueness case,” when the rank is r = n. 


There are no free variables, since n — r = 0. If there is a solution it will be the only one. 
You can see when this example has one solution or no solution: 


BA! = 


4 0 by 
x 
0 5 | | = |b is solvable exactly when b3=0. 
0 0 b3 
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A rectangular matrix cannot have both existence and uniqueness. If m is different from 
n, we cannot have r= m andr=n. 

A square matrix is the opposite. If m =n, we cannot have one property without the 
other. A square matrix has a left-inverse if and only if it has a right-inverse. There is 
only one inverse, namely B= C = A~!. Existence implies uniqueness and uniqueness 
implies existence, when the matrix is square. The condition for invertibility is full rank: 
r= m=n. Each of these conditions is a necessary and sufficient test: 


1. The columns span R”, so Ax = b has at least one solution for every b. 
2. The columns are independent, so Ax = 0 has only the solution x = 0. 


This list can be made much longer, especially if we look ahead to later chapters. Every 
condition is equivalent to every other, and ensures that A is invertible. 


3. The rows of A span R”. 

4. The rows are linearly independent. 

5. Elimination can be completed: PA = LDU, with all n pivots. 
6. The determinant of A is not zero. 

7. Zero is not an eigenvalue of A. 

8. A'A is positive definite. 


Here is a typical application to polynomials P(t) of degree n—1. The only such 
polynomial that vanishes at f),...,f, is P(t) =0. No other polynomial of degree n— 1 can 
have n roots. This is uniqueness, and it implies existence: Given any values by,...,bn, 
there exists a polynomial of degree n — 1 interpolating these values: P(t;) = b;. The 
point is that we are dealing with a square matrix; the number n of coefficients in P(t) = 
x +Xxot +--+-+x,t"~! matches the number of equations: 


Lditde Be NG by 
Interpolation lm Ho eo x2) {bo 
P(ti) =; ee : 7 

|e ee oe ee cme nl by 


That Vandermonde matrix is n by n and full rank. Ax = b always has a solution—a 
polynomial can be passed through any ); at distinct points t;. Later we shall actually find 
the determinant of A; it is not zero. 


Matrices of Rank 1 


Finally comes the easiest case, when the rank is as small as possible (except for the zero 
matrix with rank 0), One basic theme of mathematics is, given something complicated, 
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to show how it can be broken into simple pieces. For linear algebra, the simple pieces 
are matrices of rank 1: 


2 1 1 

Rank 1 A= ‘ - e has r=1 
8 4 4 
—2 -1 -l 


Every row is a multiple of the first row, so the row space is one-dimensional. In fact, we 
can write the whole matrix as the product of a column vector and a row vector: 


r i. a 

4 2 2 

A = (column) (row) See A 2 1 1. 
- = -1 


The product of a 4 by 1 matrix and a 1 by 3 matrix is a 4 by 3 matrix. This product has 
rank |. At the same time, the columns are all multiples of the same column vector; the 
column space shares the dimension r = | and reduces to a line. 


T 


Every matrix of rank 1 has the simple form A = uv’ = column times row. 


The rows are all multiples of the same vector v!, and the columns are all multiples of u. 
The row space and column space are lines—the easiest case. 


Problem Set 2.4 


1. True or false: If m =n, then the row space of A equals the column space. If m < n, 
then the nullspace has a larger dimension than : 


2. Find the dimension and construct a basis for the four subspaces associated with each 
of the matrices 


Fe ae a8 ee hese 
O: 32: 8-0) 00 0 0 


3. Find the dimension and a basis for the four fundamental subspaces for 


10. 


11. 


12. 


13. 


14. 
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Describe the four subspaces in three-dimensional space associated with 


0 1 0 
A=]0 0 1 
0 0 0 


If the product AB is the zero matrix, AB = 0, show that the column space of B is 
contained in the nullspace of A. (Also the row space of A is in the left nullspace of 
B, since each row of A multiplies B to give a zero row.) 


Suppose A is an m by n matrix of rank r. Under what conditions on those numbers 
does 


(a) A have a two-sided inverse: AA~! = A~'A = 1? 


(b) Ax = b have infinitely many solutions for every b? 
Why is there no matrix whose row space and nullspace both contain (1,1, 1)? 


Suppose the only solution to Ax = 0 (m equations in n unknowns) is x = 0. What is 
the rank and why? The columns of A are linearly , 


Find a 1 by 3 matrix whose nullspace consists of all vectors in R? such that x; + 
2x2 + 4x3 = 0. Find a 3 by 3 matrix with that same nullspace. 


If Ax = b always has at least one solution, show that the only solution to A'y = 0 is 
y =0. Hint: What is the rank? 


If Ax = 0 has a nonzero solution, show that Aly = f fails to be solvable for some 
right-hand sides f. Construct an example of A and f. 


Find the rank of A and write the matrix as A = uv!: 
100 3 
Do. =) 
A=1|00 00 and ea Ar 
20 0 6 


If a, b, c are given with a £ 0, choose d so that 


has rank 1. What are the pivots? 
Find a left-inverse and/or a right-inverse (when they exist) for 


1 0 


eae ee ee Sidhe 
O11 01 O..-@ 


A= 
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15. Ifthe columns of A are linearly independent (A is m by n), then the rank is , the 
nullspace is , the row space is , and there exists a -inverse. 


16. (A paradox) Suppose A has a right-inverse B. Then AB = J leads to A'AB = A! or 
B(A'A)~!A?. But that satisfies BA = /; it is a left-inverse. Which step is not justified? 


17. Finda matrix A that has V as its row space, and a matrix B that has V as its nullspace, 
if V is the subspace spanned by 


1 1 1 
L.3 pa ee =) 
0 0 0 


18. Find a basis for each of the four subspaces of 


OD 2 34 10 0; |}0 12 3 4 
A=!0 12 4 6;=]1 1 0} JO 0 01 2 
000 1 2 0 1 1) |0 0 0 0 0 


19. If A has the same four fundamental subspaces as B, does A = cB? 


20. (a) If a 7 by 9 matrix has rank 5, what are the dimensions of the four subspaces? 
What is the sum of all four dimensions? 


(b) If a3 by 4 matrix has rank 3, what are its column space and left nullspace? 
21. Construct a matrix with the required property, or explain why you can’t. 

(a) Column space contains H : 10) , row space contains Is | . BI : 

(b) Column space has basis B , nullspace has basis B ‘ 

(c) Dimension of nullspace = 1+ dimension of left nullspace. 


(d) Left nullspace contains [3] , row space contains Fae 


(e) Row space = column space, nullspace left nullspace. 


22. Without elimination, find dimensions and bases for the four subspaces for 


03° 3.3 1 1 
A=!0 0 0 0} and B=/4 4 
010 1 ae) 


23. Suppose the 3 by 3 matrix A is invertible. Write bases for the four subspaces for A, 
and also for the 3 by 6 matrix B= [A Al]. 


24. What are the dimensions of the four subspaces for A, B, and C, if J is the 3 by 3 


identity matrix and 0 is the 3 by 2 zero matrix? 


A=|I 0| and a= | 4 and c= ol. 


25s 


26. 


27. 


28. 


29. 


30. 


31. 
32. 


33. 
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Which subspaces are the same for these matrices of different sizes? 


(a) 4] and nr (b) 


Prove that all three matrices have the same rank r. 


A 
A 


AA 
AA 


and 


If the entries of a 3 by 3 matrix are chosen randomly between 0 and 1, what are the 
most likely dimensions of the four subspaces? What if the matrix is 3 by 5? 


(Important) A is an m by n matrix of rank r. Suppose there are right-hand sides b for 
which Ax = b has no solution. 


(a) What inequalities (< or <) must be true between m, n, and r? 


(b) How do you know that Aly = 0 has a nonzero solution? 


Construct a matrix with (1,0,1) and (1,2,0) as a basis for its row space and its 
column space. Why can’t this be a basis for the row space and nullspace? 


Without computing A, find bases for the four fundamental subspaces: 


If you exchange the first two rows of a matrix A, which of the four subspaces stay 
the same? If y = (1,2,3,4) is in the left nullspace of A, write down a vector in the 
left nullspace of the new matrix. 


Explain why v = (1,0, —1) cannot be a row of A and also be in the nullspace. 


Describe the four subspaces of R? associated with 


0 1 0 
A= |0 0 1 and J[+A= 
00 0 


oo Ff 
oD 
pe eet oS) 


(Left nullspace) Add the extra column b and reduce A to echelon form: 


1 Po a-h, to 4 Bi 
[A |= A BG PM as Oe ee sG Cppedhe 
78 9 bz 0 O 0 b3—2b.+bd, 


A combination of the rows of A has produced the zero row. What combination is it? 
(Look at b3 — 2b2 +b; on the right-hand side.) Which vectors are in the nullspace of 
A! and which are in the nullspace of A? 
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34. 


35. 


36. 


37. 


38. 


39. 


40. 


41. 


Chapter 2 Vector Spaces 


Following the method of Problem 33, reduce A to echelon form and look at zero 
rows. The b column tells which combinations you have taken of the rows: 


12 
es. as 
(a) |3 4 bol. Oa aya 
3 

46D 
2 25 by 


From the b column after elimination, read off m— r basis vectors in the left nullspace 
of A (combinations of rows that give zero). 


Suppose A is the sum of two matrices of rank one: A = uv! +wz!. 
(a) Which vectors span the column space of A? 

(b) Which vectors span the row space of A? 

(c) The rank is less than 2 if or if 

(d) Compute A and its rank if uw = z = (1,0,0) and v = w = (0,0, 1). 


Without multiplying matrices, find bases for the row and column spaces of A: 


How do you know from these shapes that A is not invertible? 
True or false (with a reason or a counterexample)? 

(a) A and A! have the same number of pivots. 

(b) A and A! have the same left nullspace. 

(c) If the row space equals the column space then A! = A. 


(d) If A! = —A then the row space of A equals the column space. 


If AB = 0, the columns of B are in the nullspace of A. If those vectors are in R”, 
prove that rank(A) + rank(B) <n. 


Can tic-tac-toe be completed (5 ones and 4 zeros in A) so that rank(A) = 2 but neither 
side passed up a winning move? 


Construct any 2 by 3 matrix of rank 1. Copy Figure 2.5 and put one vector in each 
subspace (two in the nullspace). Which vectors are orthogonal? 


Redraw Figure 2.5 for a 3 by 2 matrix of rank r= 2. Which subspace is Z (zero 
vector only)? The nullspace part of any vector x in R* is x, = 
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I am not entirely happy with the 3 by 4 matrix in the previous section. From a theoretical 
point of view it was very satisfactory; the four subspaces were computable and their 
dimensions r, n—r, r, m—r were nonzero. But the example was not produced by a 
genuine application. It did not show how fundamental those subspaces really are. 

This section introduces a class of rectangular matrices with two advantages. They 
are simple, and they are important. They are incidence matrices of graphs, and every 
entry is 1, —1, or 0. What is remarkable is that the same is true of L and U and basis 
vectors for all four subspaces. Those subspaces play a central role in network theory. 
We emphasize that the word “graph” does not refer to the graph of a function (like a 
parabola for y = x”). There is a second meaning, completely different, which is closer to 
computer science than to calculus—and it is easy to explain. This section is optional, but 
it gives a chance to see rectangular matrices in action—and how the square symmetric 
matrix A‘A turns up in the end. 

A graph consists of a set of vertices or nodes, and a set of edges that connect them. 
The graph in Figure 2.6 has 4 nodes and 5 edges. It does not have an edge between 
nodes | and 4 (and edges from a node to itself are forbidden). This graph is directed, 
because of the arrow in each edge. 

The edgenode incidence matrix is 5 by 4, with a row for every edge. If the edge goes 
from node j to node k, then that row has —1 in column j and +1 in column k. The 
incidence matrix A is shown next to the graph (and you could recover the graph if you 
only had A). Row | shows the edge from node | to node 2. Row 5 comes from the fifth 
edge, from node 3 to node 4. 


edge 1 
_ —1 ie ye 
-l1 0 1 0 
A= 0-1 1 0 
0 —-l 0 it 
= eee 0 QO -l 1 
edee? node 1 2 3 4 


Figure 2.6: A directed graph (5 edges, 4 nodes, 2 loops) and its incidence matrix A. 


Notice the columns of A. Column 3 gives information about node 3—it tells which 
edges enter and leave. Edges 2 and 3 go in, edge 5 goes out (with the minus sign). A is 
sometimes called the connectivity matrix, or the topology matrix. When the graph has m 
edges and n nodes, A is m by n (and normally m > n). Its transpose is the “node-edge” 
incidence matrix. 

Each of the four fundamental subspaces has a meaning in terms of the graph. We can 
do linear algebra, or write about voltages and currents. We do both! 
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Nullspace of A: Is there a combination of the columns that gives Ax = 0? Normally 
the answer comes from elimination, but here it comes at a glance. The columns add up 
to the zero column. The nullspace contains x = (1,1,1,1), since Ax = 0. The equation 
Ax = b does not have a unique solution (if it has a solution at all). Any “constant vector” 
x = (c,c,c,c) can be added to any particular solution of Ax = b. The complete solution 
has this arbitrary constant c (like the +C when we integrate in calculus). 

This has a meaning if we think of x1, x2, x3, x4 as the potentials (the voltages) at the 
nodes. The five components of Ax give the differences in potential across the five edges. 
The difference across edge | is x2 — x1, from the +1 in the first row. 

The equation Ax = b asks: Given the differences b;,...,b5, find the actual potentials 
X1,...,X4. But that is impossible to do! We can raise or lower all the potentials by the 
same constant c, and the differences will not change—confirming that x = (c,c,c,c) is in 
the nullspace of A. Those are the only vectors in the nullspace, since Ax = 0 means equal 
potentials across every edge. The nullspace of this incidence matrix is one-dimensional. 
The rank is 4—1 =3. 


Column Space: For which differences b;,...,b5 can we solve Ax = b? To find a 
direct test, look back at the matrix. Row 1 plus row 3 equals row 2. On the right-hand 
side we need b; + b3 = bo, or no solution is possible. Similarly, row 3 plus row 5 is row 
4. The right-hand side must satisfy b3 + bs = ba, for elimination to arrive at 0 = 0. To 
repeat, if b is in the column space, then 


bi —b2+b3=0 and b3 —ba+bs5 = 0. (1) 


Continuing the search, we also find that rows 1 +4 equal rows 2+ 5. But this is nothing 
new; subtracting the equations in (1) already produces b; +b, = by +bs5. There are 
two conditions on the five components, because the column space has dimension 5 — 2. 
Those conditions would come from elimination, but here they have a meaning on the 
graph. 

Loops: Kirchhoff’s Voltage Law says that potential differences around a loop must 
add to zero, Around the upper loop in Figure 2.6, the differences satisfy (x2 —x1) + (x3 — 
X2) = (x3 —x 1). Those differences are b; + b3 = bp. To circle the lower loop and arrive 
back at the same potential, we need b3 + bs = ba. 


2R_ The test for b to be in the column space is Kirchhoff’s Voltage Law: 


The sum of potential differences around a loop must be zero. 


Left Nullspace: To solve A'y = 0, we find its meaning on the graph. The vector y has 
five components, one for each edge. These numbers represent currents flowing along 
the five edges. Since A! is 4 by 5, the equations Ay = 0 give four conditions on those 
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five currents. They are conditions of “conservation” at each node: Flow in equals flow 
out at every node: 


Vii V5 =0 Total current to node 1 is zero 
Aty=0 yi —y3—y4 =O to node 2 
y2+y3—y5 =0 to node 3 
yatys=0 to node 4 


The beauty of network theory is that both A and A™ have important roles. 

Solving Aly = 0 means finding a set of currents that do not “pile up” at any node. The 
traffic keeps circulating, and the simplest solutions are currents around small loops. 
Our graph has two loops, and we send | amp of current around each loop: 


Loop vectors yp= [1 —1 1 0 O| and yb = |o 0 1 -1 1}. 


Each loop produces a vector y in the left nullspace. The component +1 or —1 indicates 
whether the current goes with or against the arrow. The combinations of y; and yo fill 
the left nullspace, so y; and y2 are a basis (the dimension had to be m—r=5—3 = 2). 
In fact y; — y2 = (1,—1,0,1,—1) gives the big loop around the outside of the graph. 

The column space and left nullspace are closely related. The left nullspace contains 
y; = (1,1,1,0,0), and the vectors in the column space satisfy bj — bo + b3 = 0. Then 
y'b = 0: Vectors in the column space and left nullspace are perpendicular! That is soon 
to become Part Two of the “Fundamental Theorem of Linear Algebra.” 


Row Space: The row space of A contains vectors in R*, but not all vectors. Its 
dimension is the rank r = 3. Elimination will find three independent rows, and we can 
also look to the graph. The first three rows are dependent (row 1 + row 3 = row 2, and 
those edges form a loop). Rows 1,2,4 are independent because edges 1,2,4 contain no 
loops. 

Rows 1, 2, 4 are a basis for the row space. In each row the entries add to zero. Every 
combination (f1, f2,/3, f4) in the row space will have that same property: 


finrowspace fi+/2+/3+f4=0 xinnullspace x=c(1,1,1,1) (2) 


Again this illustrates the Fundamental Theorem: The row space is perpendicular to the 
nullspace. If f is in the row space and x is in the nullspace then f'x = 0. 

For A!, the basic law of network theory is Kirchhoff’s Current Law. The total flow 
into every node is zero. The numbers f/f}, f2, 3, f4 are current sources into the nodes. The 
source f; must balance —y; — y2, which is the flow leaving node | (along edges 1 and 2). 
That is the first equation in A'y = f. Similarly at the other three nodes—conservation 
of charge requires flow in = flow out. The beautiful thing is that A! is exactly the right 
matrix for the Current Law. 


2S The equations Aly = f at the nodes express Kirchhoff’s Current Law: 
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The net current into every node is zero. Flow in = Flow out. 


This law can only be satisfied if the total current from outside is fj + f2+ 3+ 
f4 =O. With f = 0, the law Aly = 0 is satisfied by a current that goes around 
a loop. 


Spanning Trees and Independent Rows 


Every component of y; and y> in the left nullspace is 1 or —1 or 0 (from loop flows). 
The same is true of x = (1,1,1,1) in the nullspace, and all the entries in PA = LDU! The 
key point is that every elimination step has a meaning for the graph. 

You can see it in the first step for our matrix A: subtract row | from row 2. This 
replaces edge 2 by a new edge “1 minus 2”: That elimination step destroys an edge and 


edge 1 rowl -1 1 0 0 


row2 —l O 1 O 
row 1—2 Oo 1 -1 0 


creates a new edge. Here the new edge “1 — 2” is just the old edge 3 in the opposite 
direction. The next elimination step will produce zeros in row 3 of the matrix. This 
shows that rows 1, 2, 3 are dependent. Rows are dependent if the corresponding edges 
contain a loop. 

At the end of elimination we have a full set of r independent rows. Those r edges 
form a tree—a graph with no loops. Our graph has r = 3, and edges 1, 2, 4 form one 
possible tree. The full name is spanning tree because the tree “spans” all nodes of the 
graph. A spanning tree has n — 1 edges if the graph is connected, and including one more 
edge will produce a loop. 

In the language of linear algebra, n — 1 is the rank of the incidence matrix A. The row 
space has dimension n — 1. The spanning tree from elimination gives a basis for that row 
space—each edge in the tree corresponds to a row in the basis. 

The fundamental theorem of linear algebra connects the dimensions of the subspaces: 


Nullspace: dimension 1, contains x = (1,...,1). 

Column space: dimension r = n — 1, any n — 1 columns are independent. 
Row space: dimension r = n — 1, independent rows from any spanning tree. 
Left nullspace: dimension m—r=m-—n-+ 1, contains y’s from the loops. 


Those four lines give Euler’s formula, which in some way is the first theorem in topol- 
ogy. It counts zero-dimensional nodes minus one-dimensional edges plus two-dimensional 
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loops. Now it has a linear algebra proof for any connected graph: 
(# of nodes) — (# of edges) + (# of loops) = (n) —(m)+(m—n+1)=1. (3) 


For a single loop of 10 nodes and 10 edges, the Euler number is 10 — 10+ 1. If those 10 
nodes are each connected to an eleventh node in the center, then 11 — 20+ 10 is still 1. 

Every vector f in the row space has x! f = f; +--++ f, =0—the currents from outside 
add to zero. Every vector b in the column space has y'b = 0—the potential differences 
add to zero around all loops. In a moment we link x to y by a third law (Ohm’s law for 
each resistor). First we stay with the matrix A for an application that seems frivolous 
but is not. 


The Ranking of Football Teams 


At the end of the season, the polls rank college football teams. The ranking is mostly an 
average of opinions, and it sometimes becomes vague after the top dozen colleges. We 
want to rank all teams on a more mathematical basis. 

The first step is to recognize the graph. If team j played team k, there is an edge 
between them. The teams are the nodes, and the games are the edges. There are a few 
hundred nodes and a few thousand edges—which will be given a direction by an arrow 
from the visiting team to the home team. Figure 2.7 shows part of the Ivy League, 
and some serious teams, and also a college that is not famous for big time football. 
Fortunately for that college (from which I am writing these words) the graph is not 
connected. Mathematically speaking, we cannot prove that MIT is not number | (unless 
it happens to play a game against somebody). 


Harvard Yale Michigan USC Texas 
yaya i ° 
I 
] 
e I 
] 
MIT ! 
------- e 
Princeton Purdue Ohio State Notre Dame Georgia Tech 


Figure 2.7: Part of the graph for college football. 


If football were perfectly consistent, we could assign a “potential” x; to every team. 
Then if visiting team v played home team h, the one with higher potential would win. In 
the ideal case, the difference b in the score would exactly equal the difference x; — x, in 
their potentials. They wouldn’t even have to play the game! There would be complete 
agreement that the team with highest potential is the best. 

This method has two difficulties (at least). We are trying to find a number x for every 
team, and we want x; — x, = b;, for every game. That means a few thousand equations 
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and only a few hundred unknowns. The equations x, — x, = b; go into a linear system 
Ax = b, in which A is an incidence matrix. Every game has a row, with +1 in column h 
and —1 in column v—to indicate which teams are in that game. 

First difficulty: If b is not in the column space there is no solution. The scores must 
fit perfectly or exact potentials cannot be found. Second difficulty: If A has nonzero 
vectors in its nullspace, the potentials x are not well determined. In the first case x does 
not exist; in the second case x is not unique. Probably both difficulties are present. 

The nullspace always contains the vector of 1s, since A looks only at the differences 
Xp, —X,. To determine the potentials we can arbitrarily assign zero potential to Harvard. 
(I am speaking mathematically, not meanly.) But if the graph is not connected, every 
separate piece of the graph contributes a vector to the nullspace. There is even the vector 
with xyrr = | and all other x; = 0. We have to ground not only Harvard but one team 
in each piece. (There is nothing unfair in assigning zero potential; if all other potentials 
are below zero then the grounded team ranks first.) The dimension of the nullspace is 
the number of pieces of the graph—and there will be no way to rank one piece against 
another, since they play no games. 

The column space looks harder to describe. Which scores fit perfectly with a set of 
potentials? Certainly Ax = b is unsolvable if Harvard beats Yale, Yale beats Princeton, 
and Princeton beats Harvard. More than that, the score differences in that loop of games 
have to add to zero: 


Kirchhoff’s law for score differences buy + byp + bpy = 0. 


This is also a law of linear algebra. Ax = b can be solved when b satisfies the same linear 
dependencies as the rows of A. Then elimination leads to 0 = 0. 

In reality, b is almost certainly not in the column space. Football scores are not that 
consistent. To obtain a ranking we can use least squares: Make Ax as close as possible 
to b. That is in Chapter 3, and we mention only one adjustment. The winner gets a bonus 
of 50 or even 100 points on top of the score difference. Otherwise winning by | is too 
close to losing by |. This brings the computed rankings very close to the polls, and Dr. 
Leake (Notre Dame) gave a full analysis in Management Science in Sports (1976). 

After writing that subsection, I found the following in the New York Times: 


In its final rankings for 1985, the computer placed Miami (10-2) in the sev- 
enth spot above Tennessee (9-1-2). A few days after publication, packages 
containing oranges and angry letters from disgruntled Tennessee fans began 
arriving at the Times sports department. The irritation stems from the fact that 
Tennessee thumped Miami 35-7 in the Sugar Bowl. Final AP and UPI polls 
ranked Tennessee fourth, with Miami significantly lower. 


Yesterday morning nine cartons of oranges arrived at the loading dock. They 
were sent to Bellevue Hospital with a warning that the quality and contents of 
the oranges were uncertain. 
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So much for that application of linear algebra. 


Networks and Discrete Applied Mathematics 


A graph becomes a network when numbers c1,...,Cm are assigned to the edges. The 
number c; can be the length of edge i, or its capacity, or its stiffness (if it contains a 
spring), or its conductance (if it contains a resistor). Those numbers go into a diagonal 
matrix C, which is m by m. C reflects “material properties,” in contrast to the incidence 
matrix A—which gives information about the connections. 

Our description will be in electrical terms. On edge i, the conductance is c; and the 
resistance is 1 /c;. Ohm’s Law says that the current y; through the resistor is proportional 
to the voltage drop e;: 


Ohm’s Law Vi = Cie; (current) = (conductance)(voltage drop). 


This is also written E = JR, voltage drop equals current times resistance. As a vector 
equation on all edges at once, Ohm’s Law is y = Ce. 
We need Kirchhoff’s Voltage Law and Current Law to complete the framework: 


KVL: The voltage drops around each loop add to zero. 
KCL: The currents y; (and f;) into each node add to zero. 


The voltage law allows us to assign potentials x;,...,x, to the nodes. Then the dif- 
ferences around a loop give a sum like (x2 —x1) + (x3 — x2) + (x; — x3) = 0, in which 
everything cancels. The current law asks us to add the currents into each node by the 
multiplication Aly. If there are no external sources of current, Kirchhoff’s Current Law 
is Aly =0. 

The other equation is Ohm’s Law, but we need to find the voltage drop e across 
the resistor. The multiplication Ax gave the potential difference between the nodes. 
Reversing the signs, —Ax gives the drop in potential. Part of that drop may be due to a 
battery in the edge of strength b;. The rest of the drop is e = b — Ax across the resistor: 


Ohm’sLaw y=C(b—Ax) or C ly+Ax=b. (4) 


The fundamental equations of equilibrium combine Ohm and Kirchhoff into a cen- 
tral problem of applied mathematics. These equations appear everywhere: 
Cly + Ax = b 
Aly =; fe 
That is a linear symmetric system, from which e has disappeared. The unknowns are the 
currents y and the potentials x. You see the symmetric block matrix: 


os 


(5) 


Equilibrium equations 


cA 
A’ 0 


bs 
x 


Block form 
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For block elimination the pivot is C~!, the multiplier is A'C, and subtraction knocks out 
A! below the pivot. The result is 
cl oA 

0 —ATCA 


b 
f—A'Cb 


y 
x 


The equation for x alone is in the bottom row, with the symmetric matrix A'CA: 
Fundamental equation A'CAx = A'Cbt f. (7) 


Then back-substitution in the first equation produces y. Nothing mysterious—substitute 
y =C(b—Ax) into Aly = f to reach (7). 


Important Remark One potential must be fixed in advance: x, = 0. The nth node 
is grounded, and the nth column of the original incidence matrix is removed. The re- 
sulting matrix is what we now mean by A: its n— 1 columns are independent. The square 
matrix A'CA, which is the key to solving equation (7) for x, is an invertible matrix of 
order n— 1: 


| aT | C A |=| ATCA 


mxm mx (n—1) (n—1)x(n-1) 


Example 1. Suppose a battery b3 and a current source f> (and five resistors) connect 
four nodes. Node 4 is grounded and the potential x4 = 0 is fixed. The first thing is the 


en 41 Ry ry 
U5 
fo bs 
Ry Rs 
Yo 
t4 =0 
“8 Ra Ya = 
current law Aly = f at nodes 1, 2, 3: 
—y1 — ¥3 — ys = 0 —-1 0 -1 0 -!Il 
yi — yo = fo and AT=/1 -1 0 0 O 
Je oF Ba = ye OD 0 1 1 -1 0 


No equation is written for node 4, where the current law is y4+y5 + fo =0. This follows 
from adding the other three equations. 

The other equation is C-!y + Ax = b. The potentials x are connected to the currents 
y by Ohm’s Law. The diagonal matrix C contains the five conductances c; = 1/R;. The 
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right-hand side accounts for the battery of strength b3 in edge 3. The block form has 
C-l!y+Ax = b above Aly = f: 


Ri —1 1 0 YI 0 

R> O: =i a.) ja 0 

R3 —-1 0O 1 Y3 b3 

Co A ly} R4 0 O 1] Jya} — | 0 
A? O| Ix| | Rs -1 0 OJ] lys} |0 
20) ST Os 1 x 0 

1 -1 0 0 0O x2 ho 

O°. ty ale a 6 x3 0 


The system is 8 by 8, with five currents and three potentials. Elimination of y’s reduces 
to the 3 by 3 system A'CAx = A'Cb— f. The matrix A'CA contains the reciprocals 
cj = 1/R; (because in elimination you divide by the pivots). We also show the fourth 
row and column, from the grounded node, outside the 3 by 3 matrix: 


Cp bos tcs =cj —C3 —C5 (node 1) 

A'CA = —C] Cy +c2 —cC2 0 (node 2) 
—C3 —c2 cote3+c4| —cC4 (node 3) 

—C5 0 —C4 ca+c5 (node 4) 


The first entry is 1+ 1+ 1, or c; +c3+c5 when C is included, because edges 1, 3, 5 
touch node 1. The next diagonal entry is 1+ 1 or c; +c, from the edges touching node 
2. Off the diagonal the c’s appear with minus signs. The edges to the grounded node 
4 belong in the fourth row and column, which are deleted when column 4 is removed 
from A (making A'CA invertible). The 4 by 4 matrix would have all rows and columns 
adding to zero, and (1,1,1,1) would be in its nullspace. 

Notice that A'CA is symmetric. It has positive pivots and it comes from the basic 
framework of applied mathematics illustrated in Figure 2.8. 


C (Ohm’s Law) 


Figure 2.8: The framework for equilibrium: sources b and f, three steps to A'CA. 


In mechanics, x and y become displacements and stresses. In fluids, the unknowns 
are pressure and flow rate. In statistics, e is the error and x is the best least-squares fit to 
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the data. These matrix equations and the corresponding differential equations are in our 
textbook Introduction to Applied Mathematics, and the new Applied Mathematics and 
Scientific Computing. (See www.wellesleycambridge.com.) 

We end this chapter at that high point—the formulation of a fundamental problem in 
applied mathematics. Often that requires more insight than the solution of the problem. 
We solved linear equations in Chapter 1, as the first step in linear algebra. To set up the 
equations has required the deeper insight of Chapter 2. The contribution of mathematics, 
and of people, is not computation but intelligence. 


Problem Set 2.5 


1. For the 3-node triangular graph in the figure following, write the 3 by 3 incidence 
matrix A. Find a solution to Ax = 0 and describe all other vectors in the nullspace of 
A. Find a solution to Aly = 0 and describe all other vectors in the left nullspace of 
A. 


node 1 LY 
edge 1 edge 3 Y1 Y2 
node 2 edge 2 node 3 x2 Y3 X3 


2. For the same 3 by 3 matrix, show directly from the columns that every vector b in 
the column space will satisfy b; + bz — b3 = 0. Derive the same thing from the three 
rows—the equations in the system Ax = b. What does that mean about potential 
differences around a loop? 


3. Show directly from the rows that every vector f in the row space will satisfy f; + 
fo+ 3 =0. Derive the same thing from the three equations A y = f. What does that 
mean when the f’s are currents into the nodes? 


4. Compute the 3 by 3 matrix A'A, and show that it is symmetric but singular—what 
vectors are in its nullspace? Removing the last column of A (and last row of A‘) 
leaves the 2 by 2 matrix in the upper left corner; show that it is nor singular. 


5. Put the diagonal matrix C with entries c;, c2, c3 in the middle and compute ATCA, 
Show again that the 2 by 2 matrix in the upper left corner is invertible. 


6. Write the 6 by 4 incidence matrix A for the second graph in the figure. The vector 
(1,1,1,1) is in the nullspace of A, but now there will be m—n-+ 1 = 3 independent 
vectors that satisfy A'y = 0. Find three vectors y and connect them to the loops in 
the graph. 


10. 


11. 


12. 


13. 
14. 


15. 
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. If that second graph represents six games between four teams, and the score dif- 


ferences are bj,...,b6, when is it possible to assign potentials x;,...,x4 so that the 
potential differences agree with the b’s? You are finding (from Kirchhoff or from 
elimination) the conditions that make Ax = b solvable. 


. Write down the dimensions of the four fundamental subspaces for this 6 by 4 inci- 


dence matrix, and a basis for each subspace. 


. Compute A‘A and A'CA, where the 6 by 6 diagonal matrix C has entries c1,...,c6. 


How can you tell from the graph where the c’s will appear on the main diagonal of 
A'CA? 


Draw a graph with numbered and directed edges (and numbered nodes) whose inci- 
dence matrix is 


-1 1 0 0O 
—|l 

a= 0 1 O 
0 1 O -1 
0 0-1 1 


Is this graph a tree? (Are the rows of A independent?) Show that removing the last 
edge produces a spanning tree. Then the remaining rows are a basis for ? 


With the last column removed from the preceding A, and with the numbers 1. 2, 2, 1 
on the diagonal of C, write out the 7 by 7 system 


Cly + Ax = 0 
Aly =.7° 


Eliminating y;, y2, y3, y4 leaves three equations A'CAx = —f for x1, x2, x3. Solve 
the equations when f = (1,1,6). With those currents entering nodes 1, 2, 3 of the 
network, what are the potentials at the nodes and currents on the edges? 


If A is a 12 by 7 incidence matrix from a connected graph, what is its rank? How 
many free variables are there in the solution to Ax = b? How many free variables 
are there in the solution to Aly = f? How many edges must be removed to leave a 
spanning tree? 


In the graph above with 4 nodes and 6 edges, find all 16 spanning trees. 


If MIT beats Harvard 35-0, Yale ties Harvard, and Princeton beats Yale 7-6, what 
score differences in the other 3 games (H-P MIT-P, MIT-Y) will allow potential dif- 
ferences that agree with the score differences? If the score differences are known for 
the games in a spanning tree, they are known for all games. 


In our method for football rankings, should the strength of the opposition be consid- 
ered — or is that already built in? 
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16. If there is an edge between every pair of nodes (a complete graph), how many edges 
are there? The graph has n nodes, and edges from a node to itself are not allowed. 
17. For both graphs drawn below, verify Euler’s formula: 
(# of nodes) — (# of edges) + (# of loops) = 1. 


18. Multiply matrices to find A‘A, and guess how its entries come from the graph: 


(a) The diagonal of A'A tells how many into each node. 


(b) The off-diagonals —1 or 0 tell which pairs of nodes are ; 
19. Why does the nullspace of A‘A contain (1,1,1,1)? What is its rank? 


20. Why does a complete graph with n = 6 nodes have m = 15 edges? A spanning tree 
connecting all six nodes has edges. There are n”~* = 6* spanning trees! 


21. The adjacency matrix of a graph has Mj; = 1 if nodes i and j are connected by an 
edge (otherwise M;; = 0). For the graph in Problem 6 with 6 nodes and 4 edges, 
write down M and also M?. Why does (M7);; count the number of 2-step paths from 
node i to node j? 


2.6 Linear Transformations 


We know how a matrix moves subspaces around when we multiply by A. The nullspace 
goes into the zero vector. All vectors go into the column space, since Ax is always a 
combination of the columns. You will soon see something beautiful—that A takes its 
row space into its column space, and on those spaces of dimension r it is 100 percent in- 
vertible. That is the real action of A. It is partly hidden by nullspaces and left nullspaces, 
which lie at right angles and go their own way (toward zero). 

What matters now is what happens inside the space—which means inside n-dimensional 
space, if A is n by n. That demands a closer look. 

Suppose x is an n-dimensional vector. When A multiplies x, it transforms that vector 
into a new vector Ax. This happens at every point x of the n-dimensional space R”. 
The whole space is transformed, or “mapped into itself,’ by the matrix A. Figure 2.9 
illustrates four transformations that come from matrices: 
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1. A multiple of the identity matrix, A = cl, stretches every vector 

c 0 by the same factor c. The whole space expands or contracts (or 
somehow goes through the origin and out the opposite side, when 
c 1S negative). 


2. A rotation matrix turns the whole space around the origin. This 
0 || example turns all vectors through 90°, transforming every point 


(x,y) to (35%); 


3. A reflection matrix transforms every vector into its image on 
01 the opposite side of a mirror. In this example the mirror is the 
| 45° line y = x, and a point like (2,2) is unchanged. A point like 
(2, —2) is reversed to (—2,2). On a combination like v = (2,2) + 
(2,—2) = (4,0), the matrix leaves one part and reverses the other 
part. The output is Av = (2,2) + (—2,2) = (0,4) 
That reflection matrix is also a permutation matrix! It is alge- 
braically so simple, sending (x, y) to (y,x), that the geometric pic- 
ture was concealed. 


4. A projection matrix takes the whole space onto a lower- 
1 0 dimensional subspace (not invertible). The example transforms 
| each vector (x, y) in the plane to the nearest point (x,0) on the hor- 
izontal axis. That axis is the column space of A. The y-axis that 

projects to (0,0) is the nullspace. 


(cx, cy) ore 
y (ys aTs low) 
of I 7% rN 
I 
ye \ ! ‘ 
/ I \ ! 
(2,9) | 1 . 
(x,y) ! ‘X2,y) 
------ > —|—$—<—— 
stretching 90° rotation reflection (45° mirror) projection on axis 


Figure 2.9: Transformations of the plane by four matrices. 


Those examples could be lifted into three dimensions. There are matrices to stretch 
the earth or spin it or reflect it across the plane of the equator (forth pole transforming to 
south pole). There is a matrix that projects everything onto that plane (both poles to the 
center). It is also important to recognize that matrices cannot do everything, and some 
transformations T (x) are not possible with Ax: 


(i) It is impossible to move the origin, since AO = 0 for every matrix. 
(ii) If the vector x goes to x’, then 2x must go to 2x’. in general cx must go to cx’, since 
A(cx) = c(Ax). 
(iii) If the vectors x and y go to x’ and y’, then their sum x + y must go to x’ + y/—since 
A(x+y) =Ax+Ay. 
Matrix multiplication imposes those rules on the transformation. The second rule con- 
tains the first (take c = 0 to get AO = 0). We saw rule (iii) in action when (4,0) was 
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reflected across the 45° line. It was split into (2,2) + (2,—2) and the two parts were 
reflected separately. The same could be done for projections: split, project separately, 
and add the projections. These rules apply to any transformation that comes from a 
matrix. 

Their importance has earned them a name: Transformations that obey rules (i)—(111) 
are called linear transformations. The rules can be combined into one requirement: 


2T For all numbers c and d and all vectors x and y, matrix multiplication 
satisfies the rule of linearity: 


A(cx + dy) = c(Ax) + d(Ay). (1) 


Every transformation T (x) that meets this requirement is a linear transforma- 
tion. 


Any matrix leads immediately to a linear transformation. The more interesting question 
is in the opposite direction: Does every linear transformation lead to a matrix? The 
object of this section is to find the answer, yes. This is the foundation of an approach 
to linear algebra—starting with property (1) and developing its consequences—that is 
much more abstract than the main approach in this book. We preferred to begin directly 
with matrices, and now we see how they represent linear transformations. 

A transformation need not go from R” to the same space R”. It is absolutely permitted 
to transform vectors in R” to vectors in a different space R”. That is exactly what is done 
by an m by n matrix! The original vector x has n components, and the transformed vector 
Ax has m components. The rule of linearity is equally satisfied by rectangular matrices, 
so they also produce linear transformations. 

Having gone that far, there is no reason to stop. The operations in the linearity con- 
dition (1) are addition and scalar multiplication, but x and y need not be column vectors 
in R”. Those are not the only spaces. By definition, any vector space allows the com- 
binations cx + dy—the “vectors” are x and y, but they may actually be polynomials or 
matrices or functions x(t) and y(t). As long as the transformation satisfies equation (1), 
it is linear. 

We take as examples the spaces P,,, in which the vectors are polynomials p(t) of 
degree n. They look like p = aj + ajt +---+a,t", and the dimension of the vector space 
is n+ 1 (because with the constant term, there are n+ 1 coefficients). 


Example 1. The operation of differentiation, A = d /dt, is linear: 


d 
Ap(t) = a aot ait Fo + ant") =ayte+++nagt"™. (2) 


The nullspace of this A is the one-dimensional space of constants: dag/dt = 0. The 
column space is the n-dimensional space P,,_;; the right-hand side of equation (2) is 
always in that space. The sum of nullity (= 1) and rank (= n) is the dimension of the 
original space P,,. 
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Example 2. Integration from 0 to t is also linear (it takes P, to Py+1): 
Ap(t) = f (ants Fanta = agt +--+ He, (3) 
0 n+1 
This time there is no nullspace (except for the zero vector, as always!) but integration 
does not produce all polynomials in P,,,;. The right side of equation (3) has no constant 
term. Probably the constant polynomials will be the left nullspace. 


Example 3. Multiplication by a fixed polynomial like 2 + 3¢ is linear: 
Ap(t) = (2+3t)(ag + +++ + apt") = 2ag +++» +3a,t""?. 
Again this transforms P,, to P,.;, with no nullspace except p = 0. 


In these examples (and in almost all examples), linearity is not difficult to verify. It 
hardly even seems interesting. If it is there, it is practically impossible to miss. Nev- 
ertheless, it is the most important property a transformation can have!. Of course most 
transformations are not linear—for example, to square the polynomial (Ap = p7), or to 
add 1 (Ap = p +1), or to keep the positive coefficients (A(t — 17) =f). It will be linear 
transformations, and only those, that lead us back to matrices. 


Transformations Represented by Matrices 


Linearity has a crucial consequence: If we know Ax for each vector in a basis, then we 
know Ax for each vector in the entire space. Suppose the basis consists of the n vectors 
X1,.--,X,. Every other vector x is a combination of those particular vectors (they span 
the space). Then linearity determines Ax: 


Linearity If x=cyxyt-::+cpx, then Ax =c)(Ax,)+---+cy,(Axn). (4) 


The transformation T(x) = Ax has no freedom left, after it has decided what to do with 
the basis vectors. The rest is determined by linearity. The requirement (1) for two vectors 
x and y leads to condition (4) for n vectors x1,...,X,. The transformation does have a 
free hand with the vectors in the basis (they are independent). When those are settled, 
the transformation of every vector is settled. 


Example 4. What linear transformation takes x; and x2 to Ax; and Ax? 


2 4 

1 0 
xj= goesto Axy= |3]}3; m= goes to Ax2= |6 
i“ 4 : 8 


It must be multiplication T (x) = Ax by the matrix 
24 
A=|3 6 
4 8 


'Invertibility is perhaps in second place as an important property. 
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Starting with a different basis (1,1) and (2,—1), this same A is also the only linear 
transformation with 


6 0 

1 2 
Ay |= 9 and A FT lie ie 
12 0 


Next we find matrices that represent differentiation and integration. First we must 
decide on a basis. For the polynomials of degree 3 there is a natural choice for the four 
basis vectors: 


BasisforP3; pj=1, po=t, p3=t?, pat. 


That basis is not unique (it never is), but some choice is necessary and this is the most 
convenient. The derivatives of those four basis vectors are 0, 1, 2t, 3¢7: 


Action of d/dt =Ap,;=0, Ap2=pi, Ap3=2p2, Ap4=3p3. (5) 


“d/dt’ is acting exactly like a matrix, but which matrix? Suppose we were in the usual 
four-dimensional space with the usual basis—the coordinate vectors p; = (1,0,0,0), 
p2 = (0,1,0,0), p3 = (0,0, 1,0), p4 = (0,0,0, 1). The matrix is decided by equation (5): 


Differentiation matrix Aaift = 


ooo Oo 
ooo Fr 
Co OoON O&O 
oWoOeo 


Ap, 1s its first column, which is zero. Ap? is the second column, which is p;. Ap3 is 
2p2 and Apa is 3p3. The nullspace contains p; (the derivative of a constant is zero). 
The column space contains p1, p2, p3 (the derivative of a cubic is a quadratic). The 
derivative of a combination like p = 2+f—1? —?° is decided by linearity, and there is 
nothing new about that—it is the way we all differentiate. It would be crazy to memorize 
the derivative of every polynomial. 

The matrix can differentiate that p(t), because matrices build in linearity! 


0100 le 1 

1 =) 

dt 000 3{ /-1 233 
O° 500 0) ha 0 


In short, the matrix carries all the essential information. If the basis is known, and the 
matrix is known, then the transformation of every vector is known. 

The coding of the information is simple. To transform a space to itself, one basis is 
enough. A transformation from one space to another requires a basis for each. 
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2U Suppose the vectors x;,...,X, are a basis for the space V, and vectors 
Y1,--+;Ym are a basis for W. Each linear transformation T from V to W is 
represented by a matrix A. The jth column is found by applying T to the jth 
basis vector x;, and writing T(x;) as a combination of the y’s: 


Column j of A T (xj) =Axj=aijyitanjyy2t-:-+anjym. (6) 


For the differentiation matrix, column | came from the first basis vector p; = 1. Its 
derivative is zero, so column 1 is zero. The last column came from (d/dt)t? = 317. Since 
317 = 0p; +0p2+3p3+0pa, the last column contained 0, 0, 3. 0. The rule (6) constructs 
the matrix, a column at a time. 


We do the same for integration. That goes from cubics to quartics, transforming 
V = P3 into W = P4, so we need a basis for W. The natural choice is yj = 1, y2 =f, 
y3 = 07, y4 = 0°, ys = ¢*, spanning the polynomials of degree 4. The matrix A will be m 
by n, or 5 by 4. It comes from applying integration to each basis vector of V: 


t t 1 1 
fo iae=1 or Axj=y2, ..-, [ ear=34 or Ax4 = —ys. 
0 0 4 4 


Oo Oo 
oo Oo 


Integration matrix Ain = 


ooo Fr LO 
oOo ON-F 

O wir 

4 OO CO CO 


Differentiation and integration are inverse operations. Or at least integration followed 
by differentiation brings back the original function. To make that happen for matrices, 
we need the differentiation matrix from quartics down to cubics, which is 4 by 5: 


0 ° 1 
0 O 1 
Aditt = 4 and AgittAint = 

0 


Differentiation is a left-inverse of integration. Rectangular matrices cannot have two- 
sided inverses! In the opposite order, A intAqitg = 7 cannot be true. The 5 by 5 product has 
zeros in column 1. The derivative of a constant is zero. In the other columns AjnA gif iS 
the identity and the integral of the derivative of t” is t”. 
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Rotations Q, Projections P, and Reflections H 


This section began with 90° rotations, projections onto the x-axis, and reflections through 
the 45° line. Their matrices were especially simple: 


o-|? A oe a n=? | 
1 0 0 0 10 


(rotation) (projection) (reflection) 


The underlying linear transformations of the x-y plane are also simple. But rotations 
through other angles, projections onto other lines, and reflections in other mirrors are 
almost as easy to visualize, They are still linear transformations, provided that the origin 
is fixed: AO = 0. They must be represented by matrices. Using the natural basis fa and 
[9]. we want to discover those matrices. 


1. Rotation Figure 2.10 shows rotation through an angle @. It also shows the effect on 
the two basis vectors. The first one goes to (cos 8,sin@), whose length is still 1; it 
lies on the “O-line.” The second basis vector (0,1) rotates into (— sin@,cos@). By 
rule (6), those numbers go into the columns of the matrix (we use c and s for cos 0 
and sin @). This family of rotations Qg is a perfect chance to test the correspondence 
between transformations and matrices: 


Does the inverse of Qg equal Q_¢ (rotation backward through 0)? Yes. 


c —sS||c s 1 O 
Pel-e= : : < = F i 
Does the square of Qg equal Qy¢ (rotation through a double angle)? Yes. 
Oo? _ : | : 7 2 ee a a fe | . 
S| LS oe 208s G8 sin20 cos20 
Does the product of Qg and Qg equal Qo+¢ (rotation through @ then @)? 
Yes. 


cosOcosp—sinOsing ---| _ |cos(@+@) --- 
sin @ cos @+cos @ sin@ ~ | sin(@+@) 


OoQo = | 


The last case contains the first two. The inverse appears when @ is —@, and the 
square appears when @ is + @. All three questions were decided by trigonometric 
identities (and they give a new way to remember those identities). It was no accident 
that all the answers were yes. Matrix multiplication is defined exactly so that the 
product of the matrices corresponds to the product of the transformations. 


2V_ Suppose A and B are linear transformations from V to W and from U 
to V. Their product AB starts with a vector u in U, goes to Bu in V, and 
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Cc —S 
R= 

§ Cc 

C CS 
P= 

CS s? 


Figure 2.10: Rotation through @ (left). Projection onto the @-line (right). 


finishes with ABu in W. This “composition” AB is again a linear transfor- 
mation (from U to W). Its matrix is the product of the individual matrices 
representing A and B. 


For Agi¢Aint, the composite transformation was the identity (and AjptA qgi¢e annihilated 
all constants). For rotations, the order of multiplication does not matter. Then U = 
V = W is the x-y plane, and QgQo@ is the same as QgQg. For a rotation and a 
reflection, the order makes a difference. 


Technical note: To construct the matrices, we need bases for V and W, and then for 
U and V. By keeping the same basis for V, the product matrix goes correctly from 
the basis in U to the basis in W. If we distinguish the transformation A from its 
matrix (call that [A]), then the product rule 2V becomes extremely concise: [AB] = 
[A]|B]. The rule for multiplying matrices in Chapter | was totally determined by this 
requirement—it must match the product of linear transformations. 


. Projection Figure 2.10 also shows the projection of (1,0) onto the @-line. The 
length of the projection is c= cos @. Notice that the point of projection is not (c, 5), as 
I mistakenly thought; that vector has length 1 (it is the rotation), so we must multiply 
by c. Similarly the projection of (0,1) has length s, and falls at s(c,s) = (cs,s*), That 
gives the second column of the projection matrix P: 
C : 
piles 


Projection onto 0-line P= 
cs Ss 


This matrix has no inverse, because the transformation has no inverse. Points on the 
perpendicular line are projected onto the origin; that line is the nullspace of P. Points 
on the 0-line are projected to themselves! Projecting twice is the same as projecting 
once, and P? = P: 
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Image + original = 2 x projection 


Axt+a2=2Pr 


‘sof]-]-B 


Figure 2.11: Reflection through the @-line: the geometry and the matrix. 


Of course c? + s? = cos? 9 + sin? 9 = 1. A projection matrix equals its own square. 


3. Reflection Figure 2.11 shows the reflection of (1,0) in the @-line. The length of the 
reflection equals the length of the original, as it did after rotation—but here the 6- 
line stays where it is. The perpendicular line reverses direction; all points go straight 
through the mirror, Linearity decides the rest. 


I=) 2 
Reflection matrix H= | i | : 


2cs 257-1 


This matrix H has the remarkable property H* = I. Two reflections bring back 
the original. A reflection is its own inverse, H = H a which is clear from the 
geometry but less clear from the matrix. One approach is through the relationship of 
reflections to projections: H = 2P —J. This means that Hx +x = 2Px—the image 
plus the original equals twice the projection. It also confirms that H* = /: 


H? = (2P—1)*=4P?-4P+I=I, since P? =P. 


Other transformations Ax can increase the length of x; stretching and shearing are in 
the exercises. Each example has a matrix to represent it—which is the main point of this 
section. But there is also the question of choosing a basis, and we emphasize that the 
matrix depends on the choice of basis. Suppose the first basis vector is on the 0-line and 
the second basis vector is perpendicular: 


(i) The projection matrix is back to P = [4 §]. This matrix is constructed as always: 
its first column comes from the first basis vector (projected to itself). The second 
column comes from the basis vector that is projected to zero. 
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(ii) For reflections, that same basis gives H = F ae The second basis vector is re- 
flected onto its negative, to produce this second column. The matrix His still 2P—J 
when the same basis is used for H and P. 


(i11) For rotations, the matrix is not changed. Those lines are still rotated through 0, and 
Q = |¢ | as before. 


The whole question of choosing the best basis is absolutely central, and we come back 
to it in Chapter 5. The goal is to make the matrix diagonal, as achieved for P and H. To 
make Q diagonal requires complex vectors, since all real vectors are rotated. 

We mention here the effect on the matrix of a change of basis, while the linear trans- 
formation stays the same. The matrix A (or Q or P or HA) is altered to S —lAS. Thus a 
single transformation is represented by different matrices (via different bases, accounted 
for by S). The theory of eigenvectors will lead to this formula S~'AS, and to the best 
basis. 


Problem Set 2.6 


1. What matrix has the effect of rotating every vector through 90° and then projecting 
the result onto the x-axis? What matrix represents projection onto the x-axis followed 
by projection onto the y-axis? 


2. Does the product of 5 reflections and 8 rotations of the x-y plane produce a rotation 
or a reflection? 


3. The matrix A = [49] produces a stretching in the x-direction. Draw the circle x* + 
y* = 1 and sketch around it the points (2x, y) that result from multiplication by A. 
What shape is that curve? 


4. Every straight line remains straight after a linear transformation. If z is halfway 
between x and y, show that Az is halfway between Ax and Ay. 


5. The matrix A = |} 9] yields a shearing transformation, which leaves the y-axis un- 
changed. Sketch its effect on the x-axis, by indicating what happens to (1,0) and 
(2,0) and (—1,0)—and how the whole axis is transformed. 


6. What 3 by 3 matrices represent the transformations that 


(a) project every vector onto the x-y plane? 

(b) reflect every vector through the x-y plane? 

(c) rotate the x-y plane through 90°, leaving the z-axis alone? 
(d) rotate the x-y plane, then x-z, then y-z, through 90°? 


(e) carry out the same three rotations, but each one through 180°? 
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10. 


11. 
12. 


13. 


14. 
15. 


16. 


17. 
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. On the space P3 of cubic polynomials, what matrix represents d*/dt*? Construct 


the 4 by 4 matrix from the standard basis 1, t, t?, 3. Find its nullspace and column 
space. What do they mean in terms of polynomials? 


. From the cubics P3 to the fourth-degree polynomials P4, what matrix represents 


multiplication by 2+ 3r? The columns of the 5 by 4 matrix A come from applying 
the transformation to 1, f, t7, t°. 


. The solutions to the linear differential equation d*u/dt* = u form a vector space 


(since combinations of solutions are still solutions). Find two independent solutions, 
to give a basis for that solution space. 


With initial values u = x and du/dt = y at t = 0, what combination of basis vectors 
in Problem 9 solves uv” = u? This transformation from initial values to solution is 
linear. What is its 2 by 2 matrix (using x = 1, y= 0 and x = 0, y = 1 as basis for V, 
and your basis for W)? 


Verify directly from c? + s* = 1 that reflection matrices satisfy H? = 1. 


Suppose A is a linear transformation from the x-y plane to itself. Why does A~! (x + 
y) =A7!x+A7|y? If A is represented by the matrix M, explain why A~! is repre- 
sented by M~!. 


The product (AB)C of linear transformations starts with a vector x and produces 
u = Cx. Then rule 2V applies AB to u and reaches (AB)Cx. 
(a) Is this result the same as separately applying C then B then A? 


(b) Is the result the same as applying BC followed by A? Parentheses are unnecessary 
and the associative law (AB)C = A(BC) holds for linear transformations. This is 
the best proof of the same law for matrices. 


Prove that T? is a linear transformation if T is linear (from R? to R°). 


The space of all 2 by 2 matrices has the four basis “vectors” 


pa foo fa iy 


For the linear transformation of transposing, find its matrix A with respect to this 
basis. Why is A? = /? 
Find the 4 by 4 cyclic permutation matrix: (x1,x2,x3,x4) is transformed to Ax = 


(x2,.x3,2%4,X1,). What is the effect of A7? Show that A? = A~!. 


Find the 4 by 3 matrix A that represents a right shift: (x1,x2,x3) is transformed to 
(0,x1,x2,x3). Find also the /eft shift matrix B from R* back to R°, transforming 
(x1,X2,X3,X4) to (x2,x3,x4). What are the products AB and BA? 


18. 


19. 


20. 


21. 


22. 


23. 
24. 


25. 


26. 


27. 
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In the vector space P3 of all p(x) = ag + ajx + a2x* + a3x°, let S be the subset of 
polynomials with rf p(x)dx = 0. Verify that S is a subspace and find a basis. 


A nonlinear transformation is invertible if T(x) = b has exactly one solution for 
every b. The example T(x) = x? is not invertible because x” = b has two solutions 
for positive b and no solution for negative b. Which of the following transformations 
(from the real numbers R! to the real numbers R!) are invertible? None are linear, 
not even (c). 


What is the axis and the rotation angle for the transformation that takes (x1,x2,x3) 
into (x2,x3,%1)? 


A linear transformation must leave the zero vector fixed: T(0) = 0. Prove this from 
T(v+w) =T(v) +T(w) by choosing w = . Prove it also from the requirement 
T (cv) =cT(v) by choosing c = 


Which of these transformations is not linear? The input is v = (v1, v2). 


(a) T(v) =(¥2,%1). (b) T(v) = (11,%). 
(c) T(v) = (0,1). (d) T(v) = (0,1). 


If S and T are linear with S(v) = T(v) =v, then S(T(v)) = v or v?? 


Suppose 7 (v) = v, except that T(0,v2) = (0,0). Show that this transformation satis- 
fies T (cv) = cT(v) but not T(v+w) = T(v)+T(w). 


Which of these transformations satisfy T(v-+w) = T(v) + T(w), and which satisfy 
(cy) =<eT Ww)? 


(a) T(v)=v/|lv]]. (b) T(v) =v, +v2+43. 
(c) T(v) = (4,2v2, 3v3). (d) T(v) = largest component of v. 


For these transformations of V = R? to W = R’, find T(T(v)). 


(a) T(v) = —-v. 

(b) T(v) =v +(1, 1). 

(c) T(v) = 90° rotation = (—v2,v1). 

(d) T(v) = projection = ¢ — Za re). 


The “cyclic” transformation T is defined by T(vj,v2,v3) = (v2,v3,V1). What is 
T (T(T(v)))? What is T!°°(v)? 
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28. Find the range and kernel (those are new words for the column space and nullspace) 
of T. 


(a) T(v1,v2) = (v2,¥1). (b) T(v1,v2,¥3) = (1,2). 
(c) T(v1,v2) = (0,0). (d) T(11,v2) = (11,¥1). 

29. A linear transformation from V to W has an inverse from W to V when the range 
is all of W and the kernel contains only v = 0. Why are these transformations not 
invertible? 

(a) T(¥1, v2) = (v2, v2) W=R’. 
(b) T(v1,¥2) = (v1,V2,¥1 + V2) W=R>. 
(c) T(v1,v2) =v1 W=R'. 


30. Suppose a linear T transforms (1,1) to (2,2) and (2,0) to (0,0). Find T(v) when 


yy (2,2); (b) v= (3,1). (c)v=(-1,1). (d) v= (a,b). 
Problems 31-35 may be harder. The input space V contains all 2 by 2 matrices 
M, 


31. M is any 2 by 2 matrix and A = [44]. The linear transformation T is defined by 
T(M) = AM. What rules of matrix multiplication show that T is linear? 


32. Suppose A = [12]. Show that the identity matrix / is not in the range of T. Find a 
nonzero matrix M such that T(M) = AM is zero. 


33. Suppose T transposes every matrix M. Try to find a matrix A that gives AM = 
M!' for every M. Show that no matrix A will do it. To professors: Is this a linear 
transformation that doesn’t come from a matrix? 


34. The transformation 7 that transposes every matrix is definitely linear. Which of these 
extra properties are true? 
(a) T* = identity transformation. 
(b) The kernel of 7 is the zero matrix. 
(c) Every matrix is in the range of T. 
(d) T(M) = —M is impossible. 
35. Suppose T(M) = [) 8] [m] [99]. Find a matrix with T(M) # 0. Describe all ma- 


trices with T(M) = 0 (the kernel of 7) and all output matrices T(M) (the range of 
T). 


Problems 36—40 are about changing the basis 


36. (a) What matrix transforms (1,0) into (2,5) and transforms (0,1) to (1,3)? 
(b) What matrix transforms (2,5) to (1,0) and (1,3) to (0,1)? 


37. 


38. 


39. 


40. 


41. 


42. 


43. 


44. 


45. 


46. 


47. 


2.6 Linear Transformations 153 


(c) Why does no matrix transform (2,6) to (1,0) and (1,3) to (0,1)? 


(a) What matrix M transforms (1,0) and (0,1) to (7,t) and (s,u)? 
(b) What matrix N transforms (a,c) and (b,d) to (1,0) and (0,1)? 
(c) What condition on a, b, c, d will make part (b) impossible? 


(a) How do M and N in Problem 37 yield the matrix that transforms (a,c) to (7,f) 
and (b,d) to (s,u)? 
(b) What matrix transforms (2,5) to (1,1) and (1,3) to (0,2)? 


If you keep the same basis vectors but put them in a different order, the change-of- 
basis matrix M is a matrix. If you keep the basis vectors in order but change 
their lengths, M is a matrix. 


The matrix that transforms (1,0) and (0,1) to (1,4) and (1,5) is M = . The 
combination a(1,4) + b(1,5) that equals (1,0) has (a,b) = ( ). How are 
those new coordinates of (1,0) related to M or M~!? 


What are the three equations for A, B, C if the parabola Y = A+ Bx-+Cx? equals 4 
at x =a, 5 at x = b, and 6 at x = c? Find the determinant of the 3 by 3 matrix. For 
which numbers a, b, c will it be impossible to find this parabola Y? 


Suppose vj, v2, v3 are eigenvectors for T. This means T(v;) = A,v; for i = 1,2,3. 
What is the matrix for 7 when the input and output bases are the v’s? 


Every invertible linear transformation can have / as its matrix. For the output basis 
just choose w; = T(v;). Why must T be invertible? 


Suppose T is reflection across the x-axis and S is reflection across the y-axis. The 
domain V is the x-y plane. If v = (x,y) what is S(T (v))? Find a simpler description 
of the product ST. 


Suppose T is reflection across the 45° line, and S is reflection across the y-axis, If 
v = (2,1) then T(v) = (1,2). Find $(T(v)) and T(S(v)). This shows that generally 
SEATS, 

Show that the product ST of two reflections is a rotation. Multiply these reflection 
matrices to find the rotation angle: 


cos20 = sin20 cos2a@ sin2a 
sin2@ —cos20 sin2a —cos2a]}— 


The 4 by 4 Hadamard matrix is entirely +1 and —1: 


1 1 1 
Sf lt =] 
1 -1l -l 


1 
1 
H= 
1 
Loaf: if 
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Find H~! and write v = (7,5,3, 1) as a combination of the columns of H. 


Suppose we have two bases v1,...,V, and w1,...,w, for R”. If a vector has coeffi- 
cients b; in one basis and c; in the other basis, what is the change-of-basis matrix in 
b = Mc? Start from 


byvy tess + by, = Vb =cywy +--+ + CyWn = We. 


Your answer represents T(v) = v with input basis of v’s and output basis of w’s. 
Because of different bases, the matrix is not J. 


True or false: If we know T(v) for n different nonzero vectors in R’, then we know 
T (v) for every vector in R". 


(Recommended) Suppose all vectors x in the unit square 0 < x, < 1,0 <x2 < | are 
transformed to Ax (A is 2 by 2). 


(a) What is the shape of the transformed region (all Ax)? 
(b) For which matrices A is that region a square? 
(c) For which A is it a line? 


(d) For which A is the new area still 1? 


Review Exercises 


1.1 


1.2 


1.3 


Find a basis for the following subspaces of R*: 


(a) The vectors for which x1 = 2x4. 
(b) The vectors for which x; +.x2 +.x3 = 0 and x3+x4 = 0. 
(c) The subspace spanned by (1,1,1,1), (1,2,3,4), and (2,3,4,5). 


By giving a basis, describe a two-dimensional subspace of R° that contains none of 
the coordinate vectors (1,0,0), (0, 1,0), (0,0, 1). 


True or false, with counterexample if false: 


(a) If the vectors x1,...,Xm Span a subspace S, then dimS = m. 
(b) The intersection of two subspaces of a vector space cannot be empty. 
(c) If Ax = Ay, then x = y. 


(d) The row space of A has a unique basis that can be computed by reducing A to 
echelon form. 


(e) If a square matrix A has independent columns, so does A?. 


1.4 


1.5 


1.6 


1.7 
1.8 


1.9 


1.10 


1.11 
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What is the echelon form U of A? 


Po 2 Oe 2. a 
A= |-l1 —2 1 1 O 
as eS 


What are the dimensions of its four fundamental subspaces? 


Find the rank and the nullspace of 


Find bases for the four fundamental subspaces associated with 
1 2 

| RE 0 0 | Pies 1 100 . 
3 6 | oe 0101 


What is the most general solution to u+v+w=1,u—w=2? 


A= 


(a) Construct a matrix whose nullspace contains the vector x = (1, 1,2). 
(b) Construct a matrix whose left nullspace contains y = (1,5). 


(c) Construct a matrix whose column space is spanned by (1, 1,2) and whose row 
space is spanned by (1,5). 

(d) If you are given any three vectors in R° and any three vectors in R°, is there a 
6 by 5 matrix whose column space is spanned by the first three and whose row 
space is spanned by the second three? 


In the vector space of 2 by 2 matrices, 


(a) is the set of rank 1 matrices a subspace? 

(b) what subspace is spanned by the permutation matrices? 

(c) what subspace is spanned by the positive matrices (all a;; > 0)? 
(d) what subspace is spanned by the invertible matrices? 


Invent a vector space that contains all linear transformations from R” to R”. You 
have to decide on a rule for addition. What is its dimension? 


(a) Find the rank of A, and give a basis for its nullspace. 


1 1 
0 2 
0 0 
0 0 


WN NYO ke 
NO =—_- — 
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1.13 


1.14 


1.15 


1.16 
1.17 
1.18 
1.19 


1.20 


1.21 
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(b) The first 3 rows of U are a basis for the row space of A—ttrue or false? 
Columns 1, 3, 6 of U are a basis for the column space of A—true or false? 
The four rows of A are a basis for the row space of A—true or false? 

(c) Find as many linearly independent vectors b as possible for which Ax = b has a 


solution. 


(d) In elimination on A, what multiple of the third row is subtracted to knock out 
the fourth row? 


If A is an n by n—1 matrix, and its rank is n — 2, what is the dimension of its 
nullspace? 


Use elimination to find the triangular factors in A = LU, if 


> 

I 
Ss S SS 
Toor. 
oa FA 
o FF 8 


d 


Under what conditions on the numbers a, b, c, d are the columns linearly indepen- 
dent? 


Do the vectors (1, 1,3), (2,3,6), and (1,4,3) form a basis for R*? 
What do you know about C'(A) when the number of solutions to Ax = b is 


(a) 0 or 1, depending on b. 
(b) oo, independent of b. 
(c) 0 or ce, depending on b. 
(d) 1, regardless of b. 


In the previous exercise, how is r related to m and n in each example? 
If x is a vector in R", and x! y = 0 for every y, prove that x = 0. 
If A is ann by n matrix such that A* = A and rankA = n, prove that A = /. 


What subspace of 3 by 3 matrices is spanned by the elementary matrices E;;, with 
1s on the diagonal and at most one nonzero entry below? 


How many 5 by 5 permutation matrices are there? Are they linearly independent? 
Do they span the space of all 5 by 5 matrices? No need to write them all down. 


What is the rank of the n by n matrix with every entry equal to 1? How about the 
“checkerboard matrix,” with a;; = 0 when i+ j is even, aj; = 1 when i+ j is odd? 


1.22 


1.23 


1.24 


1.25 


1.26 


1.27 


1.28 


1.29 


1.30 
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(a) Ax = b has a solution under what conditions on b, for the following A and b? 


12 0 3 bi 
A=10 0 0 0 and b= |b 
240 1 b3 


(b) Find a basis for the nullspace of A. 

(c) Find the general solution to Ax = b, when a solution exists. 
(d) Find a basis for the column space of A. 

(e) What is the rank of A!? 


How can you construct a matrix that transforms the coordinate vectors e1, e2, e3 into 
three given vectors v,,v2,v3? When will that matrix be invertible? 


If e,,e2,e3 are in the column space of a 3 by 5 matrix, does it have a left-inverse? 
Does it have a right-inverse? 


Suppose T is the linear transformation on R? that takes each point (u,v,w) to (u+ 
v-+w,u-+tv,u), Describe what T~! does to the point (x,y,z). 


True or false? 


(a) Every subspace of R* is the nullspace of some matrix. 
(b) If A has the same nullspace as A!, the matrix must be square. 


(c) The transformation that takes x to mx + b is linear (from R! to R!). 


Find bases for the four fundamental subspaces of 


h 22: 3 1 
Oh 22° 2° 2 
Aj = and Ax= 1 1 4] 
0000 1 
000 4 
(a) If the rows of A are linearly independent (A is m by n) then the rank is , the 


column space is , and the left nullspace is : 


(b) If Ais 8 by 10 with a two-dimensional nullspace, show that Ax = b can be solved 
for every b. 


Describe the linear transformations of the x-y plane that are represented with stan- 
dard basis (1,0) and (0,1) by the matrices 
0 1 
; A3= : 


1 
0 om 1 0 
0 -l Ze ill 


(a) If A is square, show that the nullspace of A contains the nullspace of A. 


A\ = 
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(b) Show also that the column space of A? is contained in the column space of A. 
When does the rank-1 matrix A = uv! have A” = 0? 
(a) Find a basis for the space of all vectors in R° with x} +02 =23 +24 = ASP KE: 


(b) Find a matrix with that subspace as its nullspace. 


(c) Find a matrix with that subspace as its column space. 


Suppose the matrices in PA = LU are 
010 0; /0 0 1 -3 2 100 0/ |2 -1 4 2 1 
100 Oj |2 -1 4 LT) JOT QO) [0-0 13> 2 
0001] /4 -29 1 4) J1 11 0//0 0 0 0 2 
001 0] |}2 -1 5 -1°55 2101; {0 0 0 0 0 


(a) What is the rank of A? 

(b) What is a basis for the row space of A? 

(c) True or false: Rows 1, 2, 3 of A are linearly independent. 
(d) What is a basis for the column space of A? 

(e) What is the dimension of the left nullspace of A? 

(f) What is the general solution to Ax = 0? 


Orthogonality 


3.1. Orthogonal Vectors and Subspaces 


A basis is a set of independent vectors that span a space. Geometrically, it is a set of 
coordinate axes. A vector space is defined without those axes, but every time I think of 
the x-y plane or three-dimensional space or R”, the axes are there. They are usually per- 
pendicular! The coordinate axes that the imagination constructs are practically always 
orthogonal. In choosing a basis, we tend to choose an orthogonal basis. 

The idea of an orthogonal basis is one of the foundations of linear algebra. We need 
a basis to convert geometric constructions into algebraic calculations, and we need an 
orthogonal basis to make those calculations simple. A further specialization makes the 
basis just about optimal: The vectors should have length 1. For an orthonormal basis 
(orthogonal unit vectors), we will find 


1. the length ||x|| of a vector; 
2. the test x' y = 0 for perpendicular vectors; and 
3. how to create perpendicular vectors from linearly independent vectors. 


More than just vectors, subspaces can also be perpendicular. We will discover, so 
beautifully and simply that it will be a delight to see, that the fundamental subspaces 
meet at right angles. Those four subspaces are perpendicular in pairs, two in R” and 
two in R". That will complete the fundamental theorem of linear algebra. 

The first step is to find the length of a vector. It is denoted by ||x||, and in two 
dimensions it comes from the hypotenuse of a right triangle (Figure 3.1a). The square 
of the length was given a long time ago by Pythagoras: ||x||? = x7 +23. 


In three-dimensional space, x = (x1, x2,x3) is the diagonal of a box (Figure 3.1b). Its 
length comes from two applications of the Pythagorean formula. The two-dimensional 
case takes care of (x1,x2,0) = (1,2,0) across the base. This forms a right angle with the 
vertical side (0,0,x3) = (0,0,3). The hypotenuse of the bold triangle (Pythagoras again) 
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(0, 0, 3) SS SA SS 


7 


7 


gal 21 3) has length 14 


d 


In|? = ay + 95 + 23 
5=17+2? 


14 = 124 2? +3? 
(0, 2, 0) 


. 
s 


faescesteece e100) has length /5 
(b) 


Figure 3.1: The length of vectors (x1,x2) and (x1,%2,*3). 


(1, 0,0) 


is the length ||x|| we want: 


Length in 3D lll]? = 17 +27+3* and |lx|] = 4/x7 +23 +23. 


The extension to x = (x1,...,X,) inv dimensions is immediate. By Pythagoras n — 1 
times, the length ||x|| in R" is the positive square root of x'x: 


Length squared [acl]? = 29 tad +--+ +92 =x! x, (1) 
The sum of squares matches x!x—and the length of x = (1,2,—3) is V14: 


1 
Mx=[1 2 -3] | 2 | =1?+2?+(-3)?=14 
3 


Orthogonal Vectors 


How can we decide whether two vectors x and y are perpendicular? What is the test 
for orthogonality in Figure 3.2? In the plane spanned by x and y, those vectors are 
orthogonal provided they form a right triangle. We go back to a? +b? = c?: 


Sides of a right triangle Ill|? + Ilyl]? = |x —yl/?. (2) 
Applying the length formula (1), this test for orthogonality in R” becomes 
(hg) se Oa a) = Gr Sa) ee) 
The right-hand side has an extra —2x;y; from each (x; — y;)?: 


right-hand side = (x7 stats +x7) — 2(xiy1 + +++ +Xnyn) + (yt aes + Yr) : 
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Right angle 


=i 25 
He WE 
V5 i eee 
T tly<0 - a tly >0 
yO greater than 90° » ~~ Jess than 90° 


N 


s 


Figure 3.2: A right triangle with 5+ 20 = 25. Dotted angle 100°, dashed angle 30°. 


We have a right triangle when that sum of cross-product terms x;y; is zero: 
Orthogonal vectors xly =XxX1y1 +°++ + xy, = 0. (3) 


This sum is x! y = x;y; = y!x, the row vector x! times the column vector y: 


Inner product ity = x1 ver Xn 1] =X1yp +--+ + Xnyn.- (4) 
Yn 
This number is sometimes called the scalar product or dot product, and denoted by (x, y) 


or x-y. We will use the name inner product and keep the notation x'y. 


3A The inner product x'y is zero if and only if x and y are orthogonal vectors. 
If x'y > 0, their angle is less than 90°. If x'y < 0, their angle is greater than 
90°. 


The length squared is the inner product of x with itself: x! = x7 +--++.2x2 = ||x||?. The 
only vector with length zero—the only vector orthogonal to itself—is the zero vector. 
This vector x = 0 is orthogonal to every vector in R”. 


Example 1. (2,2,—1) is orthogonal to (—1,2,2). Both have length /4+4+4 1 =3. 
Useful fact: If nonzero vectors v,,...,v, are mutually orthogonal (every vector is 
perpendicular to every other), then those vectors are linearly independent. 


Proof. Suppose civy +--: + cyvy = 0. To show that c; must be zero, take the inner 
product of both sides with v;. Orthogonality of the v’s leaves only one term: 


vile +++ + C4V_) = civ; =0. (5) 


The vectors are nonzero, so vivy + (0 and therefore c; = 0. The same is true of every cj. 
The only combination of the v’s producing zero has all c; = 0: independence! 


The coordinate vectors e;,...,é, in R” are the most important orthogonal vectors. 
Those are the columns of the identity matrix. They form the simplest basis for R”, and 
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they are unit vectors—each has length ||e;|| = 1. They point along the coordinate axes. If 
these axes are rotated, the result is a new orthonormal basis: a new system of mutually 
orthogonal unit vectors. In R* we have cos* @ + sin? @ = 1: 


Orthonormal vectors in R? vj =(cos@,sin@) and vz =(—sin@,cos@). 


Orthogonal Subspaces 


We come to the orthogonality of two subspaces. Every vector in one subspace must be 
orthogonal to every vector in the other subspace. Subspaces of R? can have dimension 
0, 1, 2, or 3. The subspaces are represented by lines or planes through the origin— 
and in the extreme cases, by the origin alone or the whole space. The subspace {0} 
is orthogonal to all subspaces. A line can be orthogonal to another line, or it can be 
orthogonal to a plane, but a plane cannot be orthogonal to a plane. 

I have to admit that the front wall and side wall of a room look like perpendicular 
planes in R*. But by our definition, that is not so! There are lines v and w in the front 
and side walls that do not meet at a right angle. The line along the corner is in both 
walls, and it is certainly not orthogonal to itself. 


3B Two subspaces V and W of the same space R” are orthogonal if every 
vector v in V is orthogonal to every vector w in W: v'w = 0 for all v and w. 


Example 2. Suppose V is the plane spanned by v; = (1,0,0,0) and vz = (1,1,0,0). If 
W is the line spanned by w = (0,0,4,5), then w is orthogonal to both v’s. The line W 
will be orthogonal to the whole plane V. 

In this case, with subspaces of dimension 2 and 1 in R*, there is room for a third 
subspace. The line L through z = (0,0,5, —4) is perpendicular to V and W. Then the 
dimensions add to 2+ 1+ 1 =4. What space is perpendicular to all of V, W, and L? 


The important orthogonal subspaces don’t come by accident, and they come two at 
a time. In fact orthogonal subspaces are unavoidable: They are the fundamental sub- 
spaces! The first pair is the nullspace and row space. Those are subspaces of R”—the 
rows have n components and so does the vector x in Ax = 0. We have to show, using 
Ax = 0, that the rows of A are orthogonal to the nullspace vector x. 


3C Fundamental theorem of orthogonality The row space is orthogonal 
to the nullspace (in R”). The column space is orthogonal to the left nullspace 
(in R”). 


First Proof. Suppose x is a vector in the nullspace. Then Ax = 0, and this system of m 
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equations can be written out as rows of A multiplying x: 


rowl ---| |x, 0 
Every row is ed i : — 2: : 2 z : | 6) 
orthogonal to x : : : : : 

rowm --:| |Xp 0 


The main point is already in the first equation: row | is orthogonal to x. Their inner 
product is zero; that is equation 1. Every right-hand side is zero, so x is orthogonal to 
every row. Therefore x is orthogonal to every combination of the rows. Each x in the 
nullspace is orthogonal to each vector in the row space, so N(A)_LC(A!). 

The other pair of orthogonal subspaces comes from A'y = 0, or y'A = 0: 


c c 
1) O 
] ] 
yA = [v1 ee Ym hi bs =|o ae OF (7) 
n n 
1 n 


The vector y is orthogonal to every column. The equation says so, from the zeros on 
the right-hand side. Therefore y is orthogonal to every combination of the columns. 
It is orthogonal to the column space, and it is a typical vector in the left nullspace: 
N(A!)1C(A). This is the same as the first half of the theorem, with A replaced by 
Al. 


Second Proof. The contrast with this “coordinate-free proof” should be useful to the 
reader. It shows a more “abstract” method of reasoning. I wish I knew which proof is 
clearer, and more permanently understood. 

If x is in the nullspace then Ax = 0. If v is in the row space, it is a combination of the 
rows: v = A!z for some vector z. Now, in one line: 


Nullspace | Row space vix = (Alz)Ix = zlAx = 210 =0. (8) 


Example 3. Suppose A has rank 1, so its row space and column space are lines: 


1 3 
Rank-1 matrix A= |2 6 
3 9 


The rows are multiples of (1,3). The nullspace contains x = (—3, 1), which is orthogonal 
to all the rows. The nullspace and row space are perpendicular lines in R7: 


ji 3] 2]=9 and [2 6 2]=9 and [3 9) fe 


= 0. 
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In contrast, the other two subspaces are in R°. The column space is the line through 
(1,2,3). The left nullspace must be the perpendicular plane y, + 2y2 + 3y3 = 0. That 
equation is exactly the content of y'A = 0. 


The first two subspaces (the two lines) had dimensions 1 + 1 = 2 in the space R”. The 
second pair (line and plane) had dimensions 1 + 2 = 3 in the space R°. In general, the 
row space and nullspace have dimensions that add to r+ (n—r) =n. The other pair 
adds to r+ (m—r) =m. Something more than orthogonality is occurring, and I have to 
ask your patience about that one further point: the dimensions. 

It is certainly true that the null space is perpendicular to the row space—but it is not 
the whole truth. N(A) contains every vector orthogonal to the row space. The nullspace 
was formed from all solutions to Ax = 0. 


Definition. Given a subspace V of R", the space of all vectors orthogonal to V is called 
the orthogonal complement of V. It is denoted by V = “V perp.” 


Using this terminology, the nullspace is the orthogonal complement of the row space: 
N(A) =(C(A!))+. At the same time, the row space contains all vectors that are orthog- 
onal to the nullspace. A vector z can’t be orthogonal to the nullspace but outside the row 
space. Adding z as an extra row of A would enlarge the row space, but we know that 
there is a fixed formula r+ (n—r) =n: 


Dimension formula dim(row space) + dim(nullspace) = number of columns. 


Every vector orthogonal to the nullspace is in the row space: C(A!) = (N(A))t+. 

The same reasoning applied to A! produces the dual result: The left nullspace N(A‘) 
and the column space C(A) are orthogonal complements. Their dimensions add up to 
(m—r)+r=m, This completes the second half of the fundamental theorem of linear 
algebra. The first half gave the dimensions of the four subspaces. including the fact that 
row rank = column rank. Now we know that those subspaces are perpendicular. More 
than that, the subspaces are orthogonal complements. 


3D Fundamental Theorem of Linear Algebra, Part II 

The nullspace is the orthogonal complement of the row space in R”. 

The left nullspace is the orthogonal complement of the column space in R”. 
To repeat, the row space contains everything orthogonal to the nullspace. The column 
space contains everything orthogonal to the left nullspace. That is just a sentence, hidden 
in the middle of the book, but it decides exactly which equations can be solved! Looked 


at directly, Ax = b requires b to be in the column space. Looked at indirectly. Ax = b 
requires b to be perpendicular to the left nullspace. 


3E Ax = bis solvable if and only if y'b = 0 whenever y'A = 0. 
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The direct approach was “b must be a combination of the columns.” The indirect ap- 
proach is “b must be orthogonal to every vector that is orthogonal to the columns.” That 
doesn’t sound like an improvement (to put it mildly). But if only one or two vectors 
are orthogonal to the columns. it is much easier to check those one or two conditions 
y'b =0. A good example is Kirchhoff’s Voltage Law in Section 2.5. Testing for zero 
around loops is much easier than recognizing combinations of the columns. 

When the left-hand sides of Ax = b add to zero, the right-hand sides must, too: 


Xj —X2 = db, 1 -1 O 
X2 —xX3 = b2_ is solvable if and only if b} +bo2+b3=0. Here A=|]0 1 -I 
x3—-x, =b3 —-1 0 1 


This test bj + b2 +b3 = 0 makes b orthogonal to y = (1,1, 1) in the left nullspace. By 
the Fundamental Theorem, b is a combination of the columns! 


The Matrix and the Subspaces 


We emphasize that V and W can be orthogonal without being complements. Their 
dimensions can be too small. The line V spanned by (0,1,0) is orthogonal to the line 
W spanned by (0,0,1), but V is not W~. The orthogonal complement of W is a two- 
dimensional plane, and the line is only part of W~. When the dimensions are right, 
orthogonal subspaces are necessarily orthogonal complements: 


If W=V- then V=W* and dimV+dimW =n. 


In other words V++ = V. The dimensions of V and W are right, and the whole space 
R” is being decomposed into two perpendicular parts (Figure 3.3). 


Two orthogonal axes in R? Line W perpendicular to plane V 
Not orthogonal complements Orthogonal complements V = W+ 


Figure 3.3: Orthogonal complements in R*: a plane and a line (not two lines). 


Splitting R” into orthogonal parts will split every vector into x = v-+w. The vector v 
is the projection onto the subspace V. The orthogonal component w is the projection of 
x onto W. The next sections show how to find those projections of x. They lead to what 
is probably the most important figure in the book (Figure 3.4). 

Figure 3.4 summarizes the fundamental theorem of linear algebra. It illustrates the 
true effect of a matrix—what is happening inside the multiplication Ax. The nullspace 
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nullspace 


dim n—r dim m—r 


Figure 3.4: The true action Ax = A(X;ow +Xnu) Of any m by n matrix. 


is carried to the zero vector. Every Ax is in the column space. Nothing is carried to the 
left nullspace. The real action is between the row space and column space, and you see 
it by looking at a typical vector x. It has a “row space component” and a “nullspace 
component,” with x = x,;+x,. When multiplied by A, this is Ax = Ax; + AXxn: 


The nullspace component goes to zero: Ax, = 0. 


The row space component goes to the column space: Ax, = Ax. 


Of course everything goes to the column space—the matrix cannot do anything else. I 
tried to make the row and column spaces the same size, with equal dimension r. 


3F From the row space to the column space, A is actually invertible. Every 
vector b in the column space comes from exactly one vector x, in the row 
space. 


Proof. Every b in the column space is a combination Ax of the columns. In fact, b is 
Ax,, With x; in the row space, since the nullspace component gives Ax, = 0, If another 
vector x’. in the row space gives Ax}. = b, then A(x, —x’.) = b—b =0. This puts x, — x. 
in the nullspace and the row space, which makes it orthogonal to itself. Therefore it is 
zero, and x, —x/.. Exactly one vector in the row space is carried to b. 


Every matrix transforms its row space onto its column space. 


On those r-dimensional spaces A is invertible. On its nullspace A is zero. When A is 
diagonal, you see the invertible submatrix holding the r nonzeros. 

A! goes in the opposite direction, from R™ to R” and from C(A) back to C(A'). 
Of course the transpose is not the inverse! A! moves the spaces correctly, but not the 
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individual vectors. That honor belongs to A~! if it exists—and it only exists if r =m =n. 
We cannot ask A~! to bring back a whole nullspace out of the zero vector. 

When A~! fails to exist, the best substitute is the pseudoinverse At. This inverts A 
where that is possible: AT Ax = x for x in the row space. On the left nullspace, nothing 
can be done: At y = 0. Thus A* inverts A where it is invertible, and has the same rank r. 
One formula for A* depends on the singular value decomposition—for which we first 
need to know about eigenvalues. 


Problem Set 3.1 


1. Find the lengths and the inner product of x = (1,4,0,2) and y = (2, —2, 1,3). 
2. Give an example in R? of linearly independent vectors that are not orthogonal. Also, 


give an example of orthogonal vectors that are not independent. 


3. Two lines in the plane are perpendicular when the product of their slopes is —1. 
Apply this to the vectors x = (x;,x2) and y = (1,2), whose slopes are x2/x; and 
y2/y1, to derive again the orthogonality condition x'y = 0. 


4. How do we know that the ith row of an invertible matrix B is orthogonal to the jth 
column of B™!, if if j? 


5. Which pairs are orthogonal among the vectors v1, v2, v3, v4? 


6. Find all vectors in R? that are orthogonal to (1,1,1) and (1,—1,0). Produce an 
orthonormal basis from these vectors (mutually orthogonal unit vectors). 


7. Find a vector x orthogonal to the row space of A, and a vector y orthogonal to the 
column space, and a vector z orthogonal to the nullspace: 


8. If V and W are orthogonal subspaces, show that the only vector they have in common 
is the zero vector: VM W = {0}. 


9. Find the orthogonal complement of the plane spanned by the vectors (1,1,2) and 
(1,2,3), by taking these to be the rows of A and solving Ax = 0. Remember that the 
complement is a whole line. 
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10. Construct a homogeneous equation in three unknowns whose solutions are the linear 
combinations of the vectors (1, 1,2) and (1,2,3). This is the reverse of the previous 
exercise, but the two problems are really the same. 


11. The fundamental theorem is often stated in the form of Fredholm’s alternative: For 
any A and b, one and only one of the following systems has a solution: 


(i) Ax =b. 
(ii) Aly =0, y'b 40. 


Either b is in the column space C(A) or there is a y in N(A!) such that y'b 4 0. 
Show that it is contradictory for (i) and (ii) both to have solutions. 


12. Find a basis for the orthogonal complement of the row space of A: 
LQ: 2 
(ee ee Vi 


Split x = (3,3,3) into a row space component x, and a nullspace component x). 


i 


13. Illustrate the action of A’ by a picture corresponding to Figure 3.4, sending C'(A) 
back to the row space and the left nullspace to zero. 


14. Show that x — y is orthogonal to x+y if and only if ||x|| = |||]. 


15. Finda matrix whose row space contains (1,2, 1) and whose nullspace contains (1,—2, 1), 
or prove that there is no such matrix. 


16. Find all vectors that are perpendicular to (1,4,4,1) and (2,9,8,2). 


17. If V is the orthogonal complement of W in R", is there a matrix with row space V 
and nullspace W? Starting with a basis for V, construct such a matrix. 


18. If S = {0} is the subspace of R* containing only the zero vector, what is St? If S is 
spanned by (0,0,0, 1), what is S‘? What is (S+)+? 


19. Why are these statements false? 
(a) If V is orthogonal to W, then V~ is orthogonal to W~. 
(b) V orthogonal to W and W orthogonal to Z makes V orthogonal to Z. 
20. Let S be a subspace of R”. Explain what (St)+ = §S means and why it is true. 


21. Let P be the plane in R? with equation x + 2y — z= 0. Find a vector perpendicular 
to P. What matrix has the plane P as its nullspace, and what matrix has P as its row 
space? 


22. Let S be the subspace of R* containing all vectors with x; +x2-+x3+2x4 = 0. Finda 
basis for the space S~, containing all vectors orthogonal to S. 


23. 


24. 


25. 


26. 


27. 


28. 


29. 


30. 


31. 


32. 
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Construct an unsymmetric 2 by 2 matrix of rank 1. Copy Figure 3.4 and put one 
vector in each subspace. Which vectors are orthogonal? 


Redraw Figure 3.4 for a 3 by 2 matrix of rank r = 2. Which subspace is Z (zero 
vector only)? The nullspace part of any vector x in R? is x, = 


Construct a matrix with the required property or say why that is impossible. 


1 2 
(a) Column space contains 2, | and <3 , nullspace contains HE 


1 2 
(b) Row space contains 2, | and <3] , nullspace contains HE 


I 1 0 
(c) Ax = H has a solution and A! H = H ; 
(d) Every row is orthogonal to every column (A is not the zero matrix). 


(e) The columns add up to a column of Os, the rows add to a row of Is. 


If AB = 0 then the columns of B are in the of A. The rows of A are in the 
of B. Why can’t A and B be 3 by 3 matrices of rank 2? 


(a) If Ax = b has a solution and A'y = 0, then y is perpendicular to ; 


(b) If Aly = c has a solution and Ax = 0, then x is perpendicular to : 


This is a system of equations Ax = b with no solution: 


Sly Plz = 3 
20> DY Se = 5 
3x+4y+5z=9. 


Find numbers y;, y2, y3 to multiply the equations so they add to 0 = 1. You have 
found a vector y in which subspace? The inner product y"b is 1. 


In Figure 3.4, how do we know that Ax; is equal to Ax? How do we know that this 
vector is in the column space? If A = |} 1] and x = [4] what is x,? 


If Ax is in the nullspace of A! then Ax = 0. Reason: Ax is also in the of A and 
the spaces are . Conclusion: A'A has the same nullspace as A. 


Suppose A is a symmetric matrix (A! = A). 


(a) Why is its column space perpendicular to its nullspace? 


(b) If Ax = 0 and Az = 5z, which subspaces contain these “eigenvectors” x and z? 
Symmetric matrices have perpendicular eigenvectors (see Section 5.5). 


(Recommended) Draw Figure 3.4 to show each subspace for 


ies eee and en eee 
3 6 3 0 
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33. 


34. 


35. 


36. 


37. 


38. 


39. 


40. 


41. 


42. 


43. 


44. 
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Find the pieces x, and x,, and draw Figure 3.4 properly, if 


Problems 34-44 are about orthogonal subspaces. 


Put bases for the orthogonal subspaces V and W into the columns of matrices V and 
W. Why does V'W = zero matrix? This matches v' w = 0 for vectors. 


The floor and the wall are not orthogonal subspaces because they share a nonzero 
vector (along the line where they meet). Two planes in R? cannot be orthogonal! 
Find a vector in both column spaces C(A) and C(B): 


12 acd 
A=}1 3 and B= |6 3 
f° 5 1 


This will be a vector Ax and also Bx. Think 3 by 4 with the matrix [A B]. 


Extend Problem 35 to a p-dimensional subspace V and a q-dimensional subspace W 
of R”. What inequality on p+ q guarantees that V intersects W in a nonzero vector? 
These subspaces cannot be orthogonal. 


Prove that every y in N(A!) is perpendicular to every Ax in the column space, using 
the matrix shorthand of equation (8). Start from Aly = 0. 


If S is the subspace of R° containing only the zero vector, what is St? If S is spanned 
by (1,1, 1), what is St? If S is spanned by (2,0,0) and (0,0,3), what is St? 


Suppose S only contains (1,5,1) and (2,2,2) (not a subspace). Then S* is the 
nullspace of the matrix A = . S+ is a subspace even if S is not. 


Suppose L is a one-dimensional subspace (a line) in R°. Its orthogonal complement 
L+ is the perpendicular to L. Then (L+)+ is a perpendicular to L+. In 
fact (L+)+ is the same as ; 


Suppose V is the whole space R*. Then V~ contains only the vector . Then 
(V+)+ is . So (V+)+ is the same as 

Suppose S is spanned by the vectors (1,2,2,3) and (1,3,3,2). Find two vectors that 
span S~. This is the same as solving Ax = 0 for which A? 


If P is the plane of vectors in R* satisfying x; +x2+x3+x4 = 0, write a basis for 
P+. Construct a matrix that has P as its nullspace. 


If a subspace S is contained in a subspace V, prove that S+ contains V~. 


Problems 45-50 are about perpendicular columns and rows. 
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45. Suppose ann by n matrix is invertible: AA~! = J. Then the first column of A~! is 
orthogonal to the space spanned by which rows of A? 
46. Find A‘A if the columns of A are unit vectors, all mutually perpendicular. 


47. Construct a 3 by 3 matrix A with no zero entries whose columns are mutually per- 
pendicular. Compute A'A. Why is it a diagonal matrix? 


48. The lines 3x+y = b, and 6x+2y = bo are . They are the same line if .In 
that case (bj ,b2) is perpendicular to the vector . The nullspace of the matrix is 
the line 3x+ y= . One particular vector in that nullspace is : 


49. Why is each of these statements false? 


(a) (1,1, 1) is perpendicular to (1,1, —2), so the planes x+ y+ z= 0 and x+y—2z= 
0 are orthogonal subspaces. 


(b) The subspace spanned by (1,1,0,0,0) and (0,0,0,1,1) is the orthogonal com- 
plement of the subspace spanned by (1,—1,0,0,0) and (2, —2,3,4, —4). 


(c) Two subspaces that meet only in the zero vector are orthogonal. 


50. Find a matrix with v = (1,2,3) in the row space and column space. Find another 
matrix with v in the nullspace and column space. Which pairs of subspaces can v not 
be in? 


51. Suppose A is 3 by 4, B is 4 by 5, and AB = 0. Prove rank(A) + rank(B) < 4. 


52. The command N = null(A) will produce a basis for the nullspace of A. Then the 
command B = null(N’) will produce a basis for the of A. 


3.2 Cosines and Projections onto Lines 


Vectors with x'y = 0 are orthogonal. Now we allow inner products that are not zero, 
and angles that are not right angles. We want to connect inner products to angles, and 
also to transposes. In Chapter | the transpose was constructed by flipping over a matrix 
as if it were some kind of pancake. We have to do better than that. 

One fact is unavoidable: The orthogonal case is the most important. Suppose we 
want to find the distance from a point b to the line in the direction of the vector a. We 
are looking along that line for the point p closest to b. The key is in the geometry: The 
line connecting b to p (the dotted line in Figure 3.5) is perpendicular to a. This fact will 
allow us to find the projection p. Even though a and b are not orthogonal, the distance 
problem automatically brings in orthogonality. 

The situation is the same when we are given a plane (or any subspace S) instead of a 
line. Again the problem is to find the point p on that subspace that is closest to b. This 
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projection of b 


onto line through a 


Figure 3.5: The projection p is the point (on the line through a) closest to b. 


point p is the projection of b onto the subspace. A perpendicular line from b to S meets 
the subspace at p. Geometrically, that gives the distance between points b and subspaces 
S. But there are two questions that need to be asked: 


1. Does this projection actually arise in practical applications? 
2. If we have a basis for the subspace S, is there a formula for the projection p? 


The answers are certainly yes. This is exactly the problem of the least-squares solu- 
tion to an overdetermined system. The vector b represents the data from experiments 
or questionnaires, and it contains too many errors to be found in the subspace S. When 
we try to write b as a combination of the basis vectors for S, it cannot be done—the 
equations are inconsistent, and Ax = b has no solution. 

The least-squares method selects p as the best choice to replace b. There can be no 
doubt of the importance of this application. In economics and statistics, least squares 
enters regression analysis. In geodesy, the U.S. mapping survey tackled 2.5 million 
equations in 400,000 unknowns. 

A formula for p is easy when the subspace is a line. We will project b onto a in several 
different ways, and relate the projection p to inner products and angles. Projection onto a 
higher dimensional subspace is by far the most important case; it corresponds to a least- 
squares problem with several parameters, and it is solved in Section 3.3. The formulas 
are even simpler when we produce an orthogonal basis for S. 


inner products and cosines 


We pick up the discussion of inner products and angles. You will soon see that it is not 
the angle, but the cosine of the angle, that is directly related to inner products. We look 
back to trigonometry in the two-dimensional case to find that relationship. Suppose the 
vectors a and b make angles a and B with the x-axis (Figure 3.6). The length ||a|| is the 
hypotenuse in the triangle OaQ. So the sine and cosine of & are 


a| 


llal| 


P a2 
sme Tall’ cos a = 
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Figure 3.6: The cosine of the angle 9 = B — & using inner products. 


For the angle B, the sine is b2/||b|| and the cosine is b;/||b|| . The cosine of @ = B — a 

comes from an identity that no one could forget: 

a,b, +. azb2 
Ila NI 


The numerator in this formula is exactly the inner product of a and b. It gives the 
relationship between a'b and cos 6: 


(1) 


Cosine formula cos 8 = cosBcosa@+sinB sina = 


3G _ The cosine of the angle between any nonzero vectors a and b is 
a'b 


. 2 
Tall . 


Cosine of 0 cos 9 = 


This formula is dimensionally correct; if we double the length of b, then both numerator 
and denominator are doubled, and the cosine is unchanged. Reversing the sign of b, on 
the other hand, reverses the sign of cos @—and changes the angle by 180°. 

There is another law of trigonometry that leads directly to the same result. It is not so 
unforgettable as the formula in equation (1), but it relates the lengths of the sides of any 
triangle: 


Law of Cosines [|b — || = || ||? + |Ja||? —2|||||a|| cos 6. (3) 


When @ is a right angle, we are back to Pythagoras: ||b — al|* = ||b||? + |Ja||?. For any 
angle @, the expression ||b — al|* is (b—a)'(b —a), and equation (3) becomes 


b'b—2a'b+a'la=b'b+a!a—2\|b||||a\| cos 0. 


Canceling b'b and a'a on both sides of this equation, you recognize formula (2) for the 
cosine: a'b = |al|||b||cos@. In fact, this proves the cosine formula in n dimensions, 
since we only have to worry about the plane triangle Oab. 


Projection onto a Line 


Now we want to find the projection point p. This point must be some multiple p = xa of 
the given vector a—every point on the line is a multiple of a. The problem is to compute 
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O tp 
Figure 3.7: The projection p of b onto a, with cos 0 = a pe : 
Ob — |lall|lal| 


the coefficient x. All we need is the geometrical fact that the line from b to the closest 
point p = xa is perpendicular to the vector a: 
a'b 


(b—a)la, or a'(b—@)=0, or t= (4) 


That gives the formula for the number x and the projection p: 


3H The projection of the vector b onto the line in the direction of ais p = xa: 


ae ‘ = a'b 
Projection onto a line p=xa=—4. (5) 


This allows us to redraw Figure 3.5 with a correct formula for p (Figure 3.7). 

This leads to the Schwarz inequality in equation (6), which is the most important 
inequality in mathematics. A special case is the fact that arithmetic means 5 (x +y) are 
larger than geometric means ,/xy. (It is also equivalent—see Problem | at the end of 
this section—to the triangle inequality for vectors.) The Schwarz inequality seems to 
come almost accidentally from the statement that |le||” = ||b — p||? in Figure 3.7 cannot 
be negative: 


2 


—p'p-2 > 0. 


(ata) 7 


Tp Tp \2 Ty 2 Tb ala) aarp 
|p (a ie ) ata! )(aTa) — (a) 


ata ata 


This tells us that (b'b)(a'a) > (a'b)*—and then we take square roots: 


31 All vectors a and b satisfy the Schwarz inequality, which is |cos @| < 1 in 
R’: 
Schwarz inequality — [ab] < |ja||||b]|. (6) 


According to formula (2), the ratio between ab and |lal|||b|| is exactly |cos @|. Since 
all cosines lie in the interval —1 < cos@ < 1, this gives another proof of equation (6): 
the Schwarz inequality is the same as |cos@| < 1. In some ways that is a more easily 
understood proof, because cosines are so familiar. Either proof is all right in R”, but 
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notice that ours came directly from the calculation of ||b — p||?. This stays nonnegative 
when we introduce new possibilities for the lengths and inner products. The name of 
Cauchy is also attached to this inequality |a'b| < ||a||||b||, and the Russians refer to it as 
the Cauchy-Schwarz-Buniakowsky inequality! Mathematical historians seem to agree 
that Buniakowsky’s claim is genuine. 

One final observation about |a‘b| < |ja||||b||.. Equality holds if and only if b is a 
multiple of a. The angle is 80 = 0° or 8 = 180° and the cosine is 1 or —1. In this case b 
is identical with its projection p, and the distance between b and the line is zero. 


Example 1. Project b = (1,2,3) onto the line through a = (1, 1,1) to get x and p: 


T 
ab 6 
*~ ala 3 
The projection is p = xa = (2,2,2). The angle between a and b has 
IpI| _ v.12 a'b 6 
cos 8 = —— = —— andalso cos@ = a 
>| 14 llalilol| = 314 


The Schwarz inequality |a'b| < ||a||||b|| is 6 < 3/14. If we write 6 as \/36, that is the 
same as 36 < 42. The cosine is less than 1, because b is not parallel to a. 


Projection Matrix of Rank 1 


The projection of b onto the line through a lies at p = a(a'b/a'a). That is our formula 
p = xa, but it is written with a slight twist: The vector a is put before the number 
x =a'b/a'a. There is a reason behind that apparently trivial change. Projection onto 
a line is carried out by a projection matrix P, and written in this new order we can see 
what it is. P is the matrix that multiplies b and produces p: 


T aa 


ab bes Be hte 
R= ar, so the projection matrix is P= ala’ (7) 
That is a column times a row—a square matrix—divided by the number ala. 


Example 2. The matrix that projects onto the line through a = (1,1, 1) is 


1 a ie 

aa 3. 3 3 
SS Sh A, |e Arse 
ala 3 3 3 3 
1 Da, i 

3 3 3 


This matrix has two properties that we will see as typical of projections: 
1. Pis asymmetric matrix. 
2. Its square is itself: P? = P. 


P’b is the projection of Pb—and Pb is already on the line! So P*b = Pb. This matrix P 
also gives a great example of the four fundamental subspaces: 
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The column space consists of the line through a = (1,1, 1). 
The nullspace consists of the plane perpendicular to a. 
The rank is r= 1. 
Every column is a multiple of a, and so is Pb = xa. The vectors that project to p = 0 


are especially important. They satisfy a'b = 0—they are perpendicular to a and their 
component along the line is zero. They lie in the nullspace = perpendicular plane. 


Actually that example is too perfect. It has the nullspace orthogonal to the column 
space, which is haywire. The nullspace should be orthogonal to the row space. But 
because P is symmetric, its row and column spaces are the same. 


Remark on scaling The projection matrix aa'/a'a is the same if a is doubled: 


2 2 ee 
l a 43. 2 

a= |2 gives P=— |2 2 2 2| = 4 i ; as before. 
5 12 Tested 
3°33 


The line through a is the same, and that’s all the projection matrix cares about. If a has 


T T 


unit length, the denominator is a-a = 1 and the matrix is just P= aa’. 


Example 3. Project onto the “O-direction” in the x-y plane. The line goes through 
a = (cos @,sin@) and the matrix is symmetric with P* = P: 


Here c is cos@, s is sin@, and c* + s* = 1 in the denominator. This matrix P was dis- 
covered in Section 2.6 on linear transformations. Now we know P in any number of 
dimensions. We emphasize that it produces the projection p: 


To project b onto a, multiply by the projection matrix P: p = Pb. 


Transposes from Inner Products 


Finally we connect inner products to A'. Up to now, A! is simply the reflection of A 
across its main diagonal; the rows of A become the columns of A‘, and vice versa. The 
entry in row i, column j of A! is the (j,i) entry of A: 

Transpose by reflection Aij =A) 


There is a deeper significance to A!, Its close connection to inner products gives a new 
and much more “abstract” definition of the transpose: 
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3J The transpose A! can be defined by the following property: The inner 
product of Ax with y equals the inner product of x with Aly. Formally, this 
simply means that 
(Ax)'y=x"Aly=x"(A"y). (8) 
This definition gives us another (better) way to verify the formula (AB)' = B'A!, Use 
equation (8) twice: 


Move A then moveB —(ABx) 'y = (Bx) '(ATY) = x! (BTA'y). 


The transposes turn up in reverse order on the right side, just as the inverses do in the 
formula (AB)~' = B-'A~!. We mention again that these two formulas meet to give the 
remarkable combination (A~!)' = (AT)~!. 


Problem Set 3.2 


1. (a) Given any two positive numbers x and y, choose the vector b equal to (./x, JY)» 
and choose a = (J, ,/x). Apply the Schwarz inequality to compare the arith- 
metic mean 5(x+ y) with the geometric mean \/xy. 

(b) Suppose we start with a vector from the origin to the point x, and then add a 
vector of length ||y|| connecting x to x+y. The third side of the triangle goes 
from the origin to x+y. The triangle inequality asserts that this distance cannot 
be greater than the sum of the first two: 


lx +yll < [lal + III 
After squaring both sides, and expanding (x + y)'(x+y), reduce this to the 


Schwarz inequality. 
2. Verify that the length of the projection in Figure 3.7 is || p|| = ||b|| cos 8, using formula 
(5). 
3. What multiple of a = (1,1, 1) is closest to the point b = (2,4,4)? Find also the point 


closest to a on the line through b. 


4. Explain why the Schwarz inequality becomes an equality in the case that a and b 
lie on the same line through the origin, and only in that case. What if they lie on 
opposite sides of the origin? 


5. Inn dimensions, what angle does the vector (1,1,...,1) make with the coordinate 
axes? What is the projection matrix P onto that vector? 


6. The Schwarz inequality has a one-line proof if a and b are normalized ahead of time 
to be unit vectors: 


laglPee lbs? Do 
ja"b| = |) ajbj| < Y lajllbl << ; / =5 +5 = llalllel. 
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10. 


15. 
16. 


17. 
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Which previous problem justifies the middle step? 


. By choosing the correct vector b in the Schwarz inequality, prove that 


(a1 +++ +an)° <n(ay +--+ +45). 


When does equality hold? 


. The methane molecule CH, is arranged as if the carbon atom were at the center of a 


regular tetrahedron with four hydrogen atoms at the vertices. If vertices are placed 
at (0,0,0), (1,1,0), (1,0,1), and (0,1, 1)—note that all six edges have length 2, 
so the tetrahedron is regular—what is the cosine of the angle between the rays going 
from the center (5, 7 ) to the vertices? (The bond angle itself is about 109.5°, an 


old friend of chemists.) 


. Square the matrix P = aa'/a‘a, which projects onto a line, and show that P? = P. 


(Note the number a'a in the middle of the matrix aa‘aa"!) 
Is the projection matrix P invertible? Why or why not? 


(a) Find the projection matrix P; onto the line through a = [4] and also the matrix P 
that projects onto the line perpendicular to a. 


(b) Compute P; + P and P,P, and explain. 


. Find the matrix that projects every point in the plane onto the line x+ 2y = 0. 


. Prove that the trace of P = aa! /a'a—which is the sum of its diagonal entries— 


always equals 1. 


. What matrix P projects every point in R° onto the line of intersection of the planes 


x+y+t=Oandx—t=0? 
Show that the length of Ax equals the length of A'x if AAT = ATA. 
Suppose P is the projection matrix onto the line through a. 


(a) Why is the inner product of x with Py equal to the inner product of Px with y? 


(b) Are the two angles the same? Find their cosines if a = (1,1,—1), x = (2,0,1), 
y= (2,2), 

(c) Why is the inner product of Px with Py again the same? What is the angle 
between those two? 


Problems 17-26 ask for projections onto lines. Also errors e = b — p and matri- 
ces P. 


Project the vector b onto the line through a. Check that e is perpendicular to a: 


18. 


19. 


20. 


21. 


22. 


23. 


24. 


25. 
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1 1 1 —1 
(a) b= }2 and a= |1]. (b) b= }3 and a= |-—3 
2 1 1 —1 


Draw the projection of b onto a and also compute it from p = xa: 


(a) = | and o=(\] (b) =| and =|) 
sin @ 0 1 —] 


In Problem 17, find the projection matrix P = aa'/a'a onto the line through each 
vector a. Verify in both cases that P* = P. Multiply Pb in each case to compute the 
projection p. 


Construct the projection matrices P; and P onto the lines through the a’s in Problem 
18. Is it true that (P; + P,)? = P, + P:? This would be true if P,P) = 0. 


For Problems 21-26, consult the accompanying figures. 


2 Pay fi 
ag = 2 “Yess Ge 4] 
seal P, Poa, 0 
Problems 21-23 Problems 24-26 


Compute the projection matrices aa! /a'a onto the lines through aj = (—1,2,2) and 
dz = (2,2,—1), Multiply those projection matrices and explain why their product 
P,P, is what it is. 


Project b = (1,0,0) onto the lines through a; and az in Problem 21 and also onto 
a3 = (2,—1,2). Add the three projections p; + p2 + p3. 

Continuing Problems 21-22, find the projection matrix P; onto a3 = (2,—1,2). Ver- 
ify that P; + P, + P3 =I. The basis a1, a2, a3 is orthogonal! 


Project the vector b = (1,1) onto the lines through a; = (1,0) and az = (1,2). Draw 
the projections p; and p2 and add p; + po. The projections do not add to b because 
the a’s are not orthogonal. 


In Problem 24, the projection of b onto the plane of a, and a2 will equal b. Find 
P=A(A‘TA)'A? for A = [ay a2] [} 4]. 
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26. Project a; = (1,0) onto az = (1,2). Then project the result back onto a;. Draw these 
projections and multiply the projection matrices P,P): Is this a projection? 


3.3 Projections and Least Squares 


Up to this point, Ax = b either has a solution or not. If b is not in the column space C(A), 
the system is inconsistent and Gaussian elimination fails. This failure is almost certain 
when there are several equations and only one unknown: 


More equations 25: = Di 
than unknowns— 3x = Dy 
no solution? 4x = by. 


This is solvable when b1, b2, b3 are in the ratio 2:3:4. The solution x will exist only if b 
is on the same line as the column a = (2,3,4). 

In spite of their unsolvability, inconsistent equations arise all the time in practice. 
They have to be solved! One possibility is to determine x from part of the system, and 
ignore the rest; this is hard to justify if all m equations come from the same source. 
Rather than expecting no error in some equations and large errors in the others, it is 
much better to choose the x that minimizes an average error E in the m equations. 

The most convenient “‘average” comes from the sum of squares: 


Squared error = E” = (2x—b;)? + (3x— bz)? + (4x — 3)”. 


If there is an exact solution, the minimum error is FE = 0. In the more likely case that b 
is not proportional to a, the graph of E? will be a parabola. The minimum error is at the 
lowest point, where the derivative is zero: 


dE” 
dx 
Solving for x, the least-squares solution of this model system ax = b is denoted by x: 


Leastsquares solution ~  2b1+3b.+4b3 — ab 
x= _ 
: 22+4+32+442 aia 


You recognize a'b in the numerator and a‘a in the denominator. 
The general case is the same. We “solve” ax = b by minimizing 


= 2| (2x — b,)2 + (3x— b2)3 + (4x— b3)4] =0. 


E? = ||ax — b||? = (ayx— 1)? +--+» + (mx — Bm)’. 
The derivative of E* is zero at the point X, if 
(a,x — by )ay +--+ + (amX— bm )am = 0. 


We are minimizing the distance from D to the line through a, and calculus gives the same 
answer, * = (ajby +--+ +dmbm)/(aj+---+a2,), that geometry did earlier: 
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au 
: . tds AO 
3K_ The least-squares solution to a problem ax = b in one unknown is x= ——. 
a‘a 

You see that we keep coming back to the geometrical interpretation of a least-squares 
problem—to minimize a distance. By setting the derivative of E to zero, calculus con- 
firms the geometry of the previous section. The error vector e connecting b to p must be 
perpendicular to a: 


ration" b 
Orthogonality of a and e a'(b—Xa) = a'b— “—a"a = 0. 
ala 


As a side remark, notice the degenerate case a = 0. All multiples of a are zero, and 
the line is only a point. Therefore p = 0 is the only candidate for the projection. But 
the formula for x becomes a meaningless 0/0, and correctly reflects the fact that x is 
completely undetermined. All values of x give the same error E = ||Ox — b||, so E? is 
a horizontal line instead of a parabola. The “pseudoinverse” assigns the definite value 
x = 0, which is a more “symmetric” choice than any other number. 


Least Squares Problems with Several Variables 


Now we are ready for the serious step, to project b onto a subspace—trather than just 
onto a line. This problem arises from Ax = b when A is an m by n matrix. Instead 
of one column and one unknown x, the matrix now has n columns. The number m of 
observations is still larger than the number n of unknowns, so it must be expected that 
Ax = b will be inconsistent. Probably, there will not exist a choice of x that perfectly 
fits the data b. In other words, the vector b probably will not be a combination of the 
columns of A; it will be outside the column space. 

Again the problem is to choose X so as to minimize the error, and again this mini- 
mization will be done in the least-squares sense. The error is E = ||Ax — b||, and this 
is exactly the distance from b to the point Ax in the column space. Searching for the 
least-squares solution x, which minimizes E, is the same as locating the point p = Ax 
that is closer to b than any other point in the column space. 


We may use geometry or calculus to determine x. In n dimensions, we prefer the 
appeal of geometry; p must be the “projection of b onto the column space.” The error 
vector e = b— Ax must be perpendicular to that space (Figure 3.8). Finding x and the 
projection p = Ax is so fundamental that we do it in two ways: 


1. All vectors perpendicular to the column space lie in the left nullspace. Thus the 
error vector e = b— Ax must be in the nullspace of A!: 


Al(b—Ax)=0 or A'AX¥=A'D. 
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T, 

€ column a} aje =0 

| r 
ee age =0 
combine into 
Ate = Al(b— AZ) =0 


column a2 


Figure 3.8: Projection onto the column space of a 3 by 2 matrix. 


2. The error vector must be perpendicular to each column ay,...,dy of A: 
al (b—Ax) =0 at 
or b—Ax| =0. 
a‘ (b—Ax) =0 al 


This is again A'(b — AX) = 0 and ATAx = A'D, The calculus way is to take partial 
derivatives of E* = (Ax —b)'(Ax—b). That gives the same 2ATAx—2A'™b = 0. The 
fastest way is just to multiply the unsolvable equation Ax = b by A’. All these equivalent 
methods produce a square coefficient matrix A'A. It is symmetric (its transpose is not 


AA!"!) and it is the fundamental matrix of this chapter. 
The equations A'Ax = A'D are known in statistics as the normal equations. 


3L When Ax = b is inconsistent, its least-squares solution minimizes ||Ax — 
b||?: 
Normal equations A'AX=A'Db. (1) 


A'A is invertible exactly when the columns of A are linearly independent! 
Then, 
Best estimatex <= (ATA) !A'Db. (2) 


The projection of b onto the column space is the nearest point Ax: 


Projection p—=Ax=A(A'A)!A'D. (3) 


We choose an example in which our intuition is as good as the formulas: 


1 2 4 
Ax=b_ has no solution 
A=]/1 3], b= 515 oes T ; 
0 0 6 A’Ax=A-b_ gives the best x. 
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Both columns end with a zero, so C(A) is the x-y plane within three-dimensional space 
The projection of b = (4,5,6) is p = (4,5,0)—the x and y components stay the same 
but z = 6 will disappear. That is confirmed by solving the normal equations: 


LZ 5 4 
Projection peAy= |) -3 | | =a ke 
0 


In this special case, the best we can do is to solve the first two equations of Ax = b. Then 
x, = 2 and x2 = 1. The error in the equation 0x; + 0x2 = 6 is sure to be 6. 


Remark 4. Suppose b is actually in the column space of A—it is a combination b = Ax 
of the columns. Then the projection of b is still b: 


bincolumn space p=A(A!A)~'ATAx=Ax=b. 


The closest point p is just b itself—which is obvious. 
Remark 5. At the other extreme, suppose b is perpendicular to every column, so A'b = 
0. In this case b projects to the zero vector: 


bin left nullspace =p = A(A'A)~'A'H = A(ATA) 10 = 0. 


Remark 6. When A is square and invertible, the column space is the whole space. Every 
vector projects to itself, p equals b, and x = x: 


If A isinvertible = p—A(A'A)~'A'b=AA~!(A')~'AlD =D. 
This is the only case when we can take apart (A'A)~', and write itas A~'(A')~!. When 
A is rectangular that is not possible. 


Remark 7. Suppose A has only one column, containing a. Then the matrix A‘A is the 
number a'a and xis a'b/a'a. We return to the earlier formula. 


The Cross-Product Matrix A'A 


The matrix A’A is certainly symmetric. Its transpose is (A'A)' = ATA™, which is ATA 
again. Its i, 7 entry (and j, 7 entry) is the inner product of column i of A with column j 
of A. The key question is the invertibility of A'A, and fortunately 
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A'A has the same nullspace as A. 


Certainly if Ax = 0 then A'Ax = 0. Vectors x in the nullspace of A are also in the 
nullspace of ATA. To go in the other direction, start by supposing that A'Ax = 0, and 
take the inner product with x to show that Ax = 0: 


x'ATAx=0, or ||Ax|?=0, or Ax=O. 


The two nullspaces are identical. In particular, if A has independent columns (and only 
x = Ois in its nullspace), then the same is true for A'A: 


3M _ If A has independent columns, then A!A is square, symmetric, and invert- 
ible. 


We show later that A‘A is also positive definite (all pivots and eigenvalues are positive). 
This case is by far the most common and most important. Independence is not so 
hard in m-dimensional space if m > n. We assume it in what follows. 


Projection Matrices 


We have shown that the closest point to b is p = A(A'A)~'A'D. This formula expresses 
in matrix terms the construction of a perpendicular line from b to the column space of 
A. The matrix that gives p is a projection matrix, denoted by P: 


Projection matrix. P=A(A'A) ‘A’. (4) 


This matrix projects any vector b onto the column space of A.! In other words, p = Pb 
is the component of b in the column space, and the error e = b— Pb is the component 
in the orthogonal complement. (J — P is also a projection matrix! It projects b onto the 
orthogonal complement, and the projection is b — Pb.) 

In short, we have a matrix formula for splitting any b into two perpendicular compo- 
nents. Pb is in the column space C(A), and the other component (J — P)b is in the left 
nullspace N(A')—which is orthogonal to the column space. 

These projection matrices can be understood geometrically and algebraically. 


3N_ The projection matrix P = A(ATA)~!AT has two basic properties: 


(i) It equals its square: P? = P. 


(ii) It equals its transpose: P? = P. 


Conversely, any symmetric matrix with P* = P represents a projection. 


'There may be a risk of confusion with permutation matrices, also denoted by P, but the risk should be small, 
and we try never to let both appear on the same page. 
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Proof. It is easy to see why P* = P. If we start with any b, then Pb lies in the subspace 
we are projecting onto. When we project again nothing is changed. The vector Pb is 
already in the subspace, and P(PD) is still Pb. In other words P? = P. Two or three or 
fifty projections give the same point p as the first projection: 


P* = A(ATA)“!ATA(ATA) 1A? =A(A™A)“!AT =P. 


To prove that P is also symmetric, take its transpose. Multiply the transposes in 
reverse order, and use symmetry of (ATA)~!, to come back to P: 


PT =(AT)T ((ATA)"!) AT =A(ATA) AT =P 


For the converse, we have to deduce from P? = P and P! = P that Pb is the projection 
of b onto the column space of P. The error vector b — Pb is orthogonal to the space. 
For any vector Pc in the space, the inner product is zero: 


(b—Pb)'Pc =b'(I—P)'Pc =b' (P—P?)c =0. 


Thus b — Pb is orthogonal to the space, and Pb is the projection onto the column space. 


Example 1. Suppose A is actually invertible. If it is 4 by 4, then its four columns are 
independent and its column space is all of R*. What is the projection onto the whole 
space? It is the identity matrix. 


P=A(A'A)'!AT=AA1(A?) TAT =. (5) 
The identity matrix is symmetric, /* = J, and the error b — Ib is zero. 


The point of all other examples is that what happened in equation (5) is not allowed. 
To repeat: We cannot invert the separate parts A! and A when those matrices are rectan- 
gular. It is the square matrix A'A that is invertible. 


Least-Squares Fitting of Data 


Suppose we do a series of experiments, and expect the output b to be a linear function 
of the input t. We look for a straight line b = C + Dt. For example: 


1. At different times we measure the distance to a satellite on its way to Mars. In this 
case f is the time and b is the distance. Unless the motor was left on or gravity is 
strong, the satellite should move with nearly constant velocity v: b = bo + vt. 


2. We vary the load on a structure, and measure the movement it produces. In this 
experiment f is the load and D is the reading from the strain gauge. Unless the load 
is so great that the material becomes plastic, a linear relation b = C + Dt is normal 
in the theory of elasticity. 
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3. The cost of producing t books like this one is nearly linear, b = C+ Dt, with editing 
and typesetting in C and then printing and binding in D. C is the set-up cost and D 
is the cost for each additional book. 


How to compute C and D? If there is no experimental error, then two measurements 
of b will determine the line b = C+ Dt. But if there is error, we must be prepared to 
“average” the experiments and find an optimal line. That line is not to be confused with 
the line through a on which b was projected in the previous section! In fact, since there 
are two unknowns C and D to be determined, we now project onto a two-dimensional 
subspace. A perfect experiment would give a perfect C and D: 


C+ Dt = bh 
C+ Dr = bo 

| 6) 
C + Dtn = dy. 


This is an overdetermined system, with m equations and only two unknowns. If errors 
are present, it will have no solution. A has two columns, and x = (C,D): 


1 ty by 

1 ¢t C b 
5 aol. or Ayes (7) 
: | |D ; 

1 tn bin 


The best solution (C, D) is the #that minimizes the squared error E: 
Minimize = E? = ||b—Ax||* = (b} -C—Dt)* +--+» + (bm —C — Dt)’. 


The vector p = Ax is as close as possible to b. Of all straight lines b = C + Dt, we are 
choosing the one that best fits the data (Figure 3.9). On the graph, the errors are the 
vertical distances b — C — Dt to the straight line (not perpendicular distances!). It is the 
vertical distances that are squared, summed, and minimized. 


Example 2. Three measurements b, bz, b3 are marked on Figure 3.9a: 
b=1 at t=-l, BaP “on he b= 3: at r=2. 


Note that the values t = —1,1,2 are not required to be equally spaced. The first step is 
to write the equations that would hold if a line could go through all three points. Then 
every C + Dt would agree exactly with b: 


C- D= 1 1-1) 7, 1 
Ax=b is C+ D= or ees | Desh 
C + 2D = 3 | ae? 3 
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Figure 3.9: Straight-line approximation matches the projection p of b. 


If those equations Ax = b could be solved, there would be no errors. They can’t be solved 
because the points are not on a line. Therefore they are solved by least squares: 


|: 


The best solution is C — 3, D- ; and the best line is 3 + st. 


n~ 


C 


a 


ae 
2 6 


A'At—A!b is | 


Note the beautiful connections between the two figures. The problem is the same but 
the art shows it differently. In Figure 3.9b, b is not a combination of the columns (1, 1, 1) 
and (—1,1,2). In Figure 3.9, the three points are not on a line. Least squares replaces 
points b that are not on a line by points p that are! Unable to solve Ax = b, we solve 
Ax = p. 

The line 3 + st has heights 3, %, a at the measurement times —1, 1, 2. Those points 
do lie on a line. Therefore the vector p = (2, 3 7) is in the column space. This vector 
is the projection. Figure 3.9b is in three dimensions (or m dimensions if there are m 
points) and Figure 3.9a is in two dimensions (or n dimensions if there are n parameters). 

Subtracting p from b, the errors are e = (3, —§, 7): Those are the vertical errors in 
Figure 3.9a, and they are the components of the dashed vector in Figure 3.9b. This error 


vector is orthogonal to the first column (1, 1,1), since —2 — § + } = (0). It is orthogonal 


to the second column (—1, 1,2), because -5 a 4 + * = 0. It is orthogonal to the column 
space, and it is in the left nullspace. 

Question: If the measurements b = (, —§, 5) were those errors, what would be the 
best line and the best x? Answer: The zero line—which is the horizontal axis—and 
x = 0. Projection to zero. 

We can quickly summarize the equations for fitting by a straight line. The first column 
of A contains 1s, and the second column contains the times t;. Therefore A'A contains 
the sum of the 1s and the ¢; and the ie 
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30 The measurements b1,...,b are given at distinct points f,,...,fm. Then 
the straight line C + Dt which minimizes E” comes from least squares: 
C t|1C b; 
ATE] =aT» or | ™ ORAL IE) | BP 
La Ltr} |D Lub; 


Remark. The mathematics of least squares is not limited to fitting the data by straight 
lines. In many experiments there is no reason to expect a linear relationship, and it 
would be crazy to look for one. Suppose we are handed some radioactive material, The 
output b will be the reading on a Geiger counter at various times t. We may know that 
we are holding a mixture of two chemicals, and we may know their half-lives (or rates 
of decay), but we do not know how much of each is in our hands. If these two unknown 
amounts are C and D, then the Geiger counter readings would behave like the sum of 
two exponentials (and not like a straight line): 


b=Ce™ + De". (8) 
In practice, the Geiger counter is not exact. Instead, we make readings b1,...,b, at 
times f,...,fm, and equation (8) is approximately satisfied: 


Ce*t + DeH = Dd, 
Ax=b is : 
CeAtm + DeHim ws by, 

If there are more than two readings, m > 2, then in all likelihood we cannot solve for 
C and D. But the least-squares principle will give optimal values C and D. 

The situation would be completely different if we knew the amounts C and D, and 
were trying to discover the decay rates A and wu. This is a problem in nonlinear least 
squares, and it is harder. We would still form E7, the sum of the squares of the errors, 
and minimize it. But setting its derivatives to zero will not give linear equations for the 
optimal A and wu. In the exercises, we stay with linear least squares. 


Weighted Least Squares 


A simple least-squares problem is the estimate x of a patient’s weight from two obser- 
vations x = b; and x = bz. Unless b; = b2, we are faced with an inconsistent system of 


two equations in one unknown: 
1 | by 
x| = ; 
1 bo 


Up to now, we accepted b, and bo as equally reliable. We looked for the value x that 
minimized E* = (x —b;)* + (x— bp)’: 
bi +b2 


dE? 
— =0 t x= : 
dx - 2 
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The optimal ¥ is the average. The same conclusion comes from A'Ax = A'D. In fact 
A'A is a 1 by 1 matrix, and the normal equation is 2x = bj + bo. 

Now suppose the two observations are not trusted to the same degree. The value 
x = b, may be obtained from a more accurate scale—or, in a statistical problem, from a 
larger sample—than x = bz. Nevertheless, if b2 contains some information, we are not 
willing to rely totally on by. The simplest compromise is to attach different weights w7 
and W3, and choose the xw that minimizes the weighted sum of squares: 


Weighted error = E* = wi(x—b1)* + w3(x— by)’. 


If w; > wW2, more importance is attached to b;. The minimizing process (derivative = 0) 
tries harder to make (x — b,)* small: 
dE* 2 es. wid +w3b2 


— = 2[wi(x—b1) + w5(x— bo)] =0 at = xXw = 


As (9) 


wr + W5 
Instead of the average of b; and b2 (for wy = w2 = 1), xw is a weighted average of the 
data. This average is closer to b; than to bo. 

The ordinary least-squares problem leading to xy comes from changing Ax = b to 
the new system WAx = Wb. This changes the solution from x to Xw. The matrix W'W 
turns up on both sides of the weighted normal equations: 


The least squares solution to WAx = Wb is xw: 


Weighted normal equations (A'WIWA)xw = AW! Wo. 


What happens to the picture of b projected to Ax? The projection Axy is still the 
point in the column space that is closest to b. But the word “closest” has a new meaning 
when the length involves W. The weighted length of x equals the ordinary length of Wx. 
Perpendicularity no longer means y!x = 0; in the new system the test is (Wy)! (Wx) = 0. 
The matrix W'W appears in the middle. In this new sense, the projection Atw and the 
error b— Axw are again perpendicular. 

That last paragraph describes all inner products: They come from invertible matrices 
W. They involve only the symmetric combination C= W!W. The inner product of x 
and y is y'Cx. For an orthogonal matrix W = Q, when this combination is C = Q'Q = IJ, 
the inner product is not new or different. Rotating the space leaves the inner product 
unchanged. Every other W changes the length and inner product. 

For any invertible matrix W, these rules define a new inner product and length: 


Weighted byW —(x,y)w =(Wy)"(Wx) and |\x|lw =||Wxl|. (10) 


Since W is invertible, no vector is assigned length zero (except the zero vector). All 
possible inner products—which depend linearly on x and y and are positive when x = 
y # 0—are found in this way, from some matrix C = W'W. 
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In practice, the important question is the choice of C. The best answer comes from 
Statisticians, and originally from Gauss. We may know that the average error is zero. 
That is the “expected value” of the error in b—although the error is not really expected 
to be zero! We may also know the average of the square of the error; that is the variance. 
If the errors in the b; are independent of each other, and their variances are 07, then the 
right weights are w; = 1/o;. A more accurate measurement, which means a smaller 
variance, gets a heavier weight. 

In addition to unequal reliability, the observations may not be independent. If the 
errors are coupled—the polls for President are not independent of those for Senator, 
and certainly not of those for Vice-President—then W has off-diagonal terms. The best 
unbiased matrix C = W'W is the inverse of the covariance matrix—whose i, j entry is 
the expected value of (error in b;) times (error in b;). Then the main diagonal of Co} 
contains the variances o?, which are the average of (error in b;). 


Example 3. Suppose two bridge partners both guess (after the bidding) the total num- 
ber of spades they hold. For each guess, the errors —1, 0, 1 might have equal probability 
i. Then the expected error is zero and the variance is 7 
E(e) = 3(-1) + 4(0)+3(1) =0 
E(e) = 4(-1)? + 400)? + 41)? = 2. 

The two guesses are dependent, because they are based on the same bidding—but not 
identical, because they are looking at different hands. Say the chance that they are 
both too high or both too low is zero, but the chance of opposite errors is . Then 
E(e\e2) = 3(—1), and the inverse of the covariance matrix is W'W: 


=i =4 
2 1 
=o 4G 1 2 


This matrix goes into the middle of the weighted normal equations. 


E(ej) E(eiea) 
E(eyex) E(e3) 


Problem Set 3.3 


1. Find the best least-squares solution x to 3x = 10, 4x = 5. What error E” is minimized? 
Check that the error vector (10 — 3x,5 — 4x) is perpendicular to the column (3,4). 


2. Suppose the values b; = 1 and bz =7 at times t; = | and fp = 2 are fitted by a line 
b = Dt through the origin. Solve D = 1 and 2D =7 by least squares, and sketch the 
best line. 


10. 
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Solve Ax = b by least squares, and find p = Ax if 


1 O 
A= O 1 5 b= 
1 1 


a 


Verify that the error b — p is perpendicular to the columns of A. 


Write out E? = ||Ax — b]|? and set to zero its derivatives with respect to u and v, if 


u 


1 0 
A=1]0 1], :-| 
11 i 


1 
; b= |3 
4 


Compare the resulting equations with A'Ax = A‘, confirming that calculus as well 
as geometry gives the normal equations. Find the solution x and the projection p = 
Ax. Why is p = b? 


The following system has no solution: 


—1 C 4 
Ax=|1 0O =a tod eas 
D 

1 9 


Sketch and solve a straight-line fit that leads to the minimization of the quadratic 
(C—D—4)*+(C—5)?+(C+D-—9)?? What is the projection of b onto the column 
space of A? 


Find the projection of b onto the column space of A: 


ae 


Split b into p+ q, with p in the column space and q perpendicular to that space. 
Which of the four subspaces contains g? 


Find the projection matrix P onto the space spanned by a; = (1,0,1) and a) = 
(1,1,-1). 


If P is the projection matrix onto a k-dimensional subspace S of the whole space R”, 
what is the column space of P and what is its rank? 


(a) If P= P'P, show that P is a projection matrix. 
(b) What subspace does the matrix P = 0 project onto? 


If the vectors a}, a2, and b are orthogonal, what are A'A and A'b? What is the 
projection of b onto the plane of a; and a2? 
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11. 


12. 


13. 


14. 


15. 


16. 


Lie 
18. 


19. 
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Suppose P is the projection matrix onto the subspace S and Q is the projection onto 
the orthogonal complement S+. What are P+ Q and PQ? Show that P — Q is its own 
inverse. 


If V is the subspace spanned by (1,1,0,1) and (0,0, 1,0), find 


(a) a basis for the orthogonal complement V~. 
(b) the projection matrix P onto V. 
(c) the vector in V closest to the vector b = (0,1,0,—1) in V~. 


Find the best straight-line fit (least squares) to the measurements 


b=4 at t= —2, b= 3: ak t=, 
b=1 at t=O, b=0 at £=2. 
Then find the projection of b = (4,3,1,0) onto the column space of 
1 —2 
1 -1 
A= 
1 O 
de 2 


The vectors a, = (1,1,0) and a2 = (1,1,1) span a plane in R°. Find the projection 
matrix P onto the plane, and find a nonzero vector b that is projected to zero. 


If P is the projection matrix onto a line in the x-y plane, draw a figure to describe 
the effect of the “reflection matrix” H = J]—2P. Explain both geometrically and 
algebraically why H? = I. 


Show that if uw has unit length, then the rank-1 matrix P = uu! is a projection matrix: 


It has properties (i) and (ii) in 3N. By choosing u = a/||a||, P becomes the projection 
onto the line through a, and Pb is the point p = xa. Rank-1 projections correspond 
exactly to least-squares problems in one unknown. 


What 2 by 2 matrix projects the x-y plane onto the —45° line x + y = 0? 
We want to fit a plane y = C+ Dt + Ez to the four points 
y= 3, at tae) y=6 at t=0,z=3 
yea. vat 1271 y=0: at £=0;2= 0: 
(a) Find 4 equations in 3 unknowns to pass a plane through the points (if there is 
such a plane). 
(b) Find 3 equations in 3 unknowns for the best least-squares solution. 


If Py = A(A'A)~!A? is the projection onto the column space of A, what is the pro- 
jection Pr onto the row space? (It is not Pe !) 


20. 


21. 


22. 


23. 


24. 


25. 


26. 


27. 


28. 
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If P is the projection onto the column space of A, what is the projection onto the left 
nullspace? 


Suppose L; is the line through the origin in the direction of a, and L» is the line 
through b in the direction of az. To find the closest points x;a, and b+ xza on the 
two lines, write the two equations for the x; and x2 that minimize ||x;a, —x2a2 — b||. 
Solve for x if aj = (1,1,0), az = (0,1,0), b = (2, 1,4). 


Find the best line C+ Dt to fit b = 4,2, —1,0,0 at times t = —2,—1,0,1,2. 


Show that the best least-squares fit to a set of measurements y1,...,¥m by a horizontal 
line (a constant function y = C) is their average 


_ vite +Y mn 
=, 


C 


Find the best straight-line fit to the following measurements, and sketch your solu- 
tion: 


v= 2 at t—-—l, y=0 at £=0, 
y=-3 at t=, y=-5 at t=2. 


Suppose that instead of a straight line, we fit the data in Problem 24 by a parabola: 
y=C+Dt+ Et’. In the inconsistent system Ax = b that comes from the four mea- 
surements, what are the coefficient matrix A, the unknown vector x, and the data 
vector b? You need not compute x. 


A Middle-Aged man was stretched on a rack to lengths L = 5, 6, and 7 feet under 
applied forces of F = 1, 2, and 4 tons. Assuming Hooke’s law L = a+ DF, find his 
normal length a by least squares. 


Problems 27-31 introduce basic ideas of statistics—the foundation for least 


squares. 


(Recommended) This problem projects b = (b1,...,bm) onto the line through a = 
(1,...,1). We solve m equations ax = b in 1 unknown (by least squares). 

(a) Solve a'ax = a'b to show that is the mean (the average) of the b’s, 

(b) Find e = b — ax, the variance |le||?, and the standard deviation |le]|. 


(c) The horizontal line b = 3 is closest to b = (1,2,6), Check that p = (3,3,3) is 
perpendicular to e and find the projection matrix P. 


First assumption behind least squares: Each measurement error has mean zero. Mul- 
tiply the 8 error vectors b — Ax = (£1,41,+1) by (A™A)~!A! to show that the 8 
vectors x — x also average to zero. The estimate x is unbiased. 
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29. 


30. 


31. 


32. 


33. 


34. 


35. 


36. 


37. 


38. 
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Second assumption behind least squares: The m errors e; are independent with 
variance Oo”, so the average of (b — Ax)(b — Ax)! is oJ. Multiply on the left by 
(ATA)~!AT and on the right by A(A‘A)~! to show that the average of (x—x)(*— x)! 
is o?(A'A)~!. This is the all-important covariance matrix for the error in X. 


A doctor takes four readings of your heart rate. The best solution tox = b1,...,x =b4 
is the average x of bj,...,b4. The matrix A is a column of 1s. Problem 29 gives the 
expected error (x—x)? as 0?(ATA)~! = . By averaging, the variance drops from 
07 to 07/4. 

If you know the average x9 of 9 numbers bj,...,b9, how can you quickly find the 


average x19 with one more number big? The idea of recursive least squares is to 
avoid adding 10 numbers. What coefficient of x9 correctly gives x|o? 


= 0= bbwt io = (bh +-+++bj9). 


Problems 32-37 use four points b = (0,8,8,20) to bring out more ideas. 


With b = 0,8,8,20 at t = 0, 1,3, 4, set up and solve the normal equations A'AX = A!D. 
For the best straight line as in Figure 3.9a, find its four heights p; and four errors ej. 
What is the minimum value E* = e7 + e5 + e3 +7? 


(Line C+ Dt does go through p’s) With b = 0,8,8,20 at times t = 0,1,3,4, write 
the four equations Ax = b (unsolvable). Change the measurements to p = 1,5, 13,17 
and find an exact solution to Ax = p. 


Check that e = b— p = (—1,3,—5,3) is perpendicular to both columns of A. What 
is the shortest distance |le|| from b to the column space of A? 


For the closest parabola b = C + Dt + Et’ to the same four points, write the unsolv- 
able equations Ax = b in three unknowns x = (C,D,E). Set up the three normal 
equations A'Ax = Ab (solution not required). You are now fitting a parabola to four 
points—what is happening in Figure 3.9b? 


For the closest cubic b = C++ Dt + Et? + FP’ to the same four points, write the four 
equations Ax = b. Solve them by elimination, This cubic now goes exactly through 
the points. What are p and e? 


The average of the four times is ?= (0+ 1+3+4) =2. The average of the four b’s 
is b = 1(0+8+8+20) =9. 


(a) Verify that the best line goes through the center point (7, b) = (2,9), 
(b) Explain why C+ Dt = b comes from the first equation in ATAX = A'D. 


What happens to the weighted average xj = (w7b1 + w3b2)/(w7 +w3) if the first 
weight w, approaches zero? The measurement b, is totally unreliable. 
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39. Fromm independent measurements b,,...,b,, of your pulse rate, weighted by w1,...,Wm, 
what is the weighted average that replaces equation (9)? It is the best estimate when 
the statistical variances are 07 = 1/w?. 


40. If W = [49], find the W-inner product of x = (2,3) and y = (1, 1), and the W-length 
of x. What line of vectors is W-perpendicular to y? 


41. Find the weighted least-squares solution xw to Ax = b: 


1 0 0 20 0 
A=1l1 1 b= 1/1 W=]0 1 0 
1 2 1 00 1 


Check that the projection Axy is still perpendicular (in the W-inner product!) to the 
error b— Ax. 


42. (a) Suppose you guess your professor’s age, making errors e = —2,—1,5 with prob- 
abilities $,1,1. Check that the expected error E(e) is zero and find the variance 
5 2°44 p 
E(e*). 


(b) If the professor guesses too (or tries to remember), making errors —1, 0, 1 with 


“14° 1 6 1 
probabilities 3, 3,3, 


and the professor’s guess? 


what weights w; and w2 give the reliability of your guess 


3.4 Orthogonal Bases and Gram-Schmidt 


In an orthogonal basis, every vector is perpendicular to every other vector. The coor- 
dinate axes are mutually orthogonal. That is just about optimal, and the one possible 
improvement is easy: Divide each vector by its length, to make it a unit vector. That 
changes an orthogonal basis into an orthonormal basis of q’s: 


3P The vectors gj,...,¢, are orthonormal if 


tT. _ JO whenever i ¥ j, giving the orthogonality; 
4i4i~ 1 whenever i = I; giving the normalization. 


A matrix with orthonormal columns will be called QO. 


The most important example is the standard basis. For the x-y plane, the best-known 
axes € = (1,0) and e2 = (0,1) are not only perpendicular but horizontal and vertical. Q 
is the 2 by 2 identity matrix. In n dimensions the standard basis e1,...,é,) again consists 
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of the columns of Q =I: 


1 0 0 
Standard : : : 
on re ey = 0 » &2= 0 yp ty On = 0 
basis 


i) 
— 


That is not the only orthonormal basis! We can rotate the axes without changing the 
right angles at which they meet. These rotation matrices will be examples of Q. 

If we have a subspace of R”, the standard vectors e; might not lie in that subspace. 
But the subspace always has an orthonormal basis, and it can be constructed in a simple 
way out of any basis whatsoever. This construction, which converts a skewed set of axes 
into a perpendicular set, is known as Gram-Schmidt orthogonalization. 

To summarize, the three topics basic to this section are: 


1. The definition and properties of orthogonal matrices Q. 
2. The solution of Qx = b, either n by n or rectangular (least squares). 


3. The Gram-Schmidt process and its interpretation as a new factorization A = QR. 


Orthogonal Matrices 


3Q_ If Q (square or rectangular) has orthonormal columns, then Q'Q = I: 


— q 1 0 0 

Orthonormal Ez g— || | [-% 20 
columns : hE P| ees ae 

3 gt kL. i | or 


(1) 


An orthogonal matrix is a square matrix with orthonormal columns.” Then 
Q' is Q~!. For square orthogonal matrices, the transpose is the inverse. 


When row i of QO! multiplies column j of Q, the result is qj j = 0. On the diagonal 
where i = j, we have q) qi = |. That is the normalization to unit vectors of length 1. 
Note that Q'Q =/ even if Q is rectangular. But then Q! is only a left-inverse. 


Example 1. 


O= Fe | oh'=o"'= ee a 


sin@Q cos@ —sin@ cos@ 


?Orthonormal matrix would have been a better name, but it is too late to change. Also, there is no accepted word 
for a rectangular matrix with orthonormal columns. We still write Q, but we won’t call it an “orthogonal matrix” 
unless it is square. 
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Q rotates every vector through the angle 6, and Q! rotates it back through —@. The 
columns are clearly orthogonal, and they are orthonormal because sin*@ + cos*@ = 1. 
The matrix Q! is just as much an orthogonal matrix as Q. 


Example 2. Any permutation matrix P is an orthogonal matrix. The columns are cer- 
tainly unit vectors and certainly orthogonal—because the | appears in a different place 
in each column: The transpose is the inverse. 


010 001 
If P=|0 01] thn P'=P'’=1]1 00 
100 010 


An anti-diagonal P, with P;3 = Po2 = P3; =I, takes the x-y-z axes into the z-y-x axes— 
a “right-handed” system into a “left-handed” system. So we were wrong if we suggested 
that every orthogonal Q represents a rotation. A reflection is also allowed. P = {9 }| 
reflects every point (x,y) into (y,x), its mirror image across the 45° line. Geometrically, 


an orthogonal Q is the product of a rotation and a reflection. 


There does remain one property that is shared by rotations and reflections, and in fact 
by every orthogonal matrix. It is not shared by projections, which are not orthogonal or 
even invertible. Projections reduce the length of a vector, whereas orthogonal matrices 
have a property that is the most important and most characteristic of all: 


3R Multiplication by any Q preserves lengths: 
Lengths unchanged ||Qx|| = ||x|| for every vector x. (2) 


It also preserves inner products and angles, since (Qx)'(Qy) = x'Q' Oy =x'y. 


The preservation of lengths comes directly from Q'Q = I: 
||Ox||? = |||? because (Qx)"(Qx) =x"Q" Ox =x"x. (3) 


All inner products and lengths are preserved, when the space is rotated or reflected. 

We come now to the calculation that uses the special property Q' = Q~'. If we have a 
basis, then any vector is a combination of the basis vectors. This is exceptionally simple 
for an orthonormal basis, which will be a key idea behind Fourier series. The problem 
is to find the coefficients of the basis vectors: 


Write b as a combination b= x\q\ +x2G2 +--:+XnGn- 


To compute x; there is a neat trick. Multiply both sides of the equation by ats On the 
left-hand side is qib. On the right-hand side all terms disappear (because qig j = 0) 
except the first term. We are left with 


qib=x1q, 1- 
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Since giq, = 1, we have found x, = q1b. Similarly the second coefficient is x. = q4b; 
that term survives when we multiply by q+. The other terms die of orthogonality. Each 
piece of b has a simple formula, and recombining the pieces gives back b: 


Every vector b is equal to (qi b)qi + (q3b)qo + +--+ (qhb) an: (4) 


I can’t resist putting this orthonormal basis into a square matrix Q. The vector equa- 
tion x1g1 +::-+Xngn = b is identical to Qx = b. (The columns of Q multiply the compo- 
nents of x.) Its solution is x = Q~'b. But since Q~! = Q'—this is where orthonormality 
enters—the solution is also x = Q'b: 


— gq — qib 
£=0' b= bl =|: (5) 
— gq — gb 


The components of x are the inner products qi b, as in equation (4). 

The matrix form also shows what happens when the columns are not orthonormal. 
Expressing b as a combination x;a; +---+xydy is the same as solving Ax = b. The basis 
vectors go into the columns of A. In that case we need A~!, which takes work. In the 
orthonormal case we only need Q'. 


Remark 1. The ratio a'b/a‘a appeared earlier, when we projected b onto a line. Here a 
is qi, the denominator is 1, and the projection is (q{b)q1. Thus we have a new interpre- 
tation for formula (4): Every vector b is the sum of its one-dimensional projections onto 
the lines through the q’s. 

Since those projections are orthogonal, Pythagoras should still be correct. The square 
of the hypotenuse should still be the sum of squares of the components: 


|? = (a1b)" + (43b) +--+ (Gnb)? whichis ||Q*D||?. (6) 


Remark 2. Since Q' = Q~!, we also have QQ! =]. When Q comes before Q', mul- 
tiplication takes the inner products of the rows of Q. (For Q'Q it was the columns.) 
Since the result is again the identity matrix, we come to a surprising conclusion: The 
rows of a square matrix are orthonormal whenever the columns are. The rows point 
in completely different directions from the columns, and I don’t see geometrically why 
they are forced to be orthonormal—but they are. 


1/V3 1/v2 1/v6 
Q=|1/V3 0 —-2/V6]. 
Orthonormal rows tae BU Yaa /V6 


Orthonormal columns 


Rectangular Matrices with Orthogonal Columns 


This chapter is about Ax = b, when A is not necessarily square. For Qx = b we now 
admit the same possibility—there may be more rows than columns. The 7 orthonormal 
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vectors qj; in the columns of Q have m > n components. Then Q is an m by n matrix and 
we cannot expect to solve Qx = b exactly. We solve it by least squares. 

If there is any justice, orthonormal columns should make the problem simple. It 
worked for square matrices, and now it will work for rectangular matrices. The key is to 
notice that we still have Q'O = 1. So Q' is still the left-inverse of Q. 

For least squares that is all we need. The normal equations came from multiplying 
Ax = b by the transpose matrix, to give A'Ax = A'b. Now the normal equations are 
O'Q=Q'b. But Q'Q is the identity matrix! Therefore ¥ = Q'b, whether Q is square 
and x is an exact solution, or Q is rectangular and we need least squares. 


3S If Q has orthonormal columns, the least-squares problem becomes easy: 
rectangular system with no solution for most b. 


Ox = b rectangular system with no solution for most b. 
O'ox = Q'b normal equation for the best x—in which Q'Q = I. 
x = Q'b x; is qi b. 
p = OX the projection of b is (qib)qi +---+(qib)qn- 
p QO'b the projection matrix is P= QO". 


The last formulas are like p = Ax and P= A(A'A)~'A™. When the columns are orthonor- 
mal, the “cross-product matrix” A'A becomes Q'Q = J. The hard part of least squares 
disappears when vectors are orthonormal. The projections onto the axes are uncoupled, 
and p is the sum p = (q}b)qi +---+ (qld) qn. 

We emphasize that those projections do not reconstruct b. In the square case m = n, 
they did. In the rectangular case m > n, they don’t. They give the projection p and 
not the original vector b 
than unknowns, and the q’s are no longer a basis. The projection matrix is usually 
A(ATA)~!Al, and here it simplifies to 


which is all we can expect when there are more equations 


P=Q(0'0)'Q" or P=Q@Q"'. (7) 


Notice that Q'Q is the n by n identity matrix, whereas QQ! is an m by m projection P. 
It is the identity matrix on the columns of Q (P leaves them alone), But QQ! is the zero 
matrix on the orthogonal complement (the nullspace of Q"). 


Example 3. The following case is simple but typical. Suppose we project a point 
b = (x,y,z) onto the x-y plane. Its projection is p = (x,y,0), and this is the sum of the 
separate projections onto the x- and y-axes: 


1 x 0 0 
gi=|0| and (qglb)q:= |01; qz=|1) and (q3b)qo= Jy 
0 0 0 0 
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The overall projection matrix is 


1 0 0 x Ba 
P=qqi't+qqh=|0 1 0], and Ply| =]y 
000 z| {0 


Projection onto a plane = sum of projections onto orthonormal q, and qp. 


Example 4. When the measurement times average to zero, fitting a straight line leads to 
orthogonal columns. Take t; = —3, tg = 0, and fz = 3. Then the attempt to fit y =C+ Dt 
leads to three equations in two unknowns: 


C + Dt = yy 1 -—3 yy 
C + Dt = yo, or 1 O D = yo 
C + Dth = y3 1 3 y3 


The columns (1,1,1) and (—3,0,3) are orthogonal. We can project y separately onto 
each column, and the best coefficients C and D can be found separately: 


i. T 
Z 1 1 1 Iv y2 v3) : 3 0 3] ly y2 v3) 
a eee de hts (3) 0 3- 

Notice that C = (y, + y2 + y3)/3 is the mean of the data. C gives the best fit by a 

horizontal line, whereas Dr is the best fit by a straight line through the origin. The 

columns are orthogonal, so the sum of these two separate pieces is the best fit by any 

straight line whatsoever. The columns are not unit vectors, so C and D have the length 

squared in the denominator. 


Orthogonal columns are so much better that it is worth changing to that case. if the 
average of the observation times is not zero—it is f = (ft; +---+t)/m—then the time 
origin can be shifted by f. Instead of y = C+ Dt we work with y =c+d(t —f). The best 
line is the same! As in the example, we find 


«ea prea) Sees, 
ta Yeti m % 
pend mA] fn om Seay 

(1-7 + + (im 8? yi? 


The best C is the mean, and we also get a convenient formula for d. The earlier ATA 
had the off-diagonal entries )'t;, and shifting the time by ¢ made these entries zero. This 
shift is an example of the Gram-Schmidt process, which orthogonalizes the situation in 
advance. 

Orthogonal matrices are crucial to numerical linear algebra, because they introduce 
no instability. While lengths stay the same, roundoff is under control. Orthogonalizing 
vectors has become an essential technique. Probably it comes second only to elimina- 
tion. And it leads to a factorization A = QR that is nearly as famous as A = LU. 
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The Gram-Schmidt Process 


Suppose you are given three independent vectors a, b, c. If they are orthonormal, life is 
easy. To project a vector v onto the first one, you compute (a'v)a. To project the same 
vector v onto the plane of the first two, you just add (a'v)a+ (b'v)b. To project onto 
the span of a, b, c, you add three projections. All calculations require only the inner 
products a'yv, b'y, and c'v. But to make this true, we are forced to say, “If they are 
orthonormal.” Now we propose to find a way to make them orthonormal. 

The method is simple. We are given a, b, c and we want q1, g2, q3. There is no 
problem with q;: it can go in the direction of a. We divide by the length, so that gq; = 
a/||a|| is a unit vector. The real problem begins with gz—which has to be orthogonal 
to q,. If the second vector b has any component in the direction of g; (which is the 
direction of a), that component has to be subtracted: 


Second vector B=b—(qjb)qi and q7=B/|B\. (9) 


B is orthogonal to q. It is the part of b that goes in a new direction, and not in the a. In 
Figure 3.10, B is perpendicular to q,. It sets the direction for qo. 


Figure 3.10: The g; component of b is removed; a and B normalized to q; and q2. 


At this point gq; and q2 are set. The third orthogonal direction starts with c. It will 
not be in the plane of g; and q2, which is the plane of a and b. However, it may have a 
component in that plane, and that has to be subtracted. (If the result is C = 0, this signals 
that a, b, c were not independent in the first place) What is left is the component C we 
want, the part that is in a new direction perpendicular to the plane: 


Third vector C=c—(qic)qi—(qoc)q2 and q3=C/||C\|. (10) 


This is the one idea of the whole Gram-Schmidt process, to subtract from every new 
vector its components in the directions that are already settled. That idea is used over 
and over again.? When there is a fourth vector, we subtract away its components in the 
directions of g1, 92, g3. 


3If Gram thought of it first, what was left for Schmidt? 
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Example 5. Gram-Schmidt Suppose the independent vectors are a, b, c: 


1 1 2 
a=!/0|, b=|0|, c=/1 
1 0 0 


To find g;, make the first vector into a unit vector: g; = a/ /2. To find q2, subtract from 
the second vector its component in the first direction: 


1 a |e V2 Ne 
B=b—(qib)qi = 0 mv) 0 = 0 
0 1/2 —1 
The normalized gz is B divided by its length, to produce a unit vector: 
1/2 
q2 = 0 : 
—1//2 


To find q3, subtract from c its components along q; and qo: 


C=c-(qic)qi—(qc)a 


2 1/V2 1/V2 0 
=|1}-v2]} 0 |-v2} 0 J=/1 
0 1/2 —1//2 0 


This is already a unit vector, so it is g3. I went to desperate lengths to cut down the num- 
ber of square roots (the painful part of Gram-Schmidt). The result is a set of orthonormal 
vectors g1, 92, 73, Which go into the columns of an orthogonal matrix Q: 


1/J/2 1/V/2 0 
Orthonormal basis Q=!0 @= @l|= 0 0 1 
1/V/2 -1/V2 0 
3T The Gram-Schmidt process starts with independent vectors a1,...,a, and 
ends with orthonormal vectors q1,...,gn. At step j it subtracts from a; its 
components in the directions qgj,...,qj—1 that are already settled: 
Aj =aj—(qtaj)qi — +++ — (4j_-14))4j-1- (11) 


Then q; is the unit vector A ;/||Aj||. 


Remark on the calculations {| think it is easier to compute the orthogonal a, B, C, 
without forcing their lengths to equal one. Then square roots enter only at the end, when 
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dividing by those lengths. The example above would have the same B and C, without 
using square roots. Notice the 5 from a'b/a'a instead of a from q'b: 


il), - are 2 1 5 
B= |0 ae 0} andthen C= |{1] —|0} —2] 0 
0 1 0 1 —5 


The Factorization A = OR 


We started with a matrix A, whose columns were a, b, c. We ended with a matrix Q, 
whose columns are g1, g2, g3. What is the relation between those matrices? The matrices 
A and Q are m by n when the n vectors are in m-dimensional space, and there has to be 
a third matrix that connects them. 

The idea is to write the a’s as combinations of the qg’s. The vector b in Figure 3.10 is 
a combination of the orthonormal g; and g2, and we know what combination it is: 


b = (qib)qi + (q3b)q2. 


Every vector in the plane is the sum of its g; and gz components. Similarly c is the sum 
of its qi, q2, g3 components: c = (qic)qi + (q3c)q2 + (q4c)q3. If we express that in 
matrix form we have the new factorization A = OR: 


qja qb ajc 
ORfactors A=l]a b cl=|q mq gib qic| =QR_ (12) 

q3c 
Notice the zeros in the last matrix! R is upper triangular because of the way Gram- 
Schmidt was done. The first vectors a and q fell on the same line. Then qg1, g2 were in 

the same plane as a, b. The third vectors c and q3 were not involved until step 3. 

The QR factorization is like A = LU, except that the first factor Q has orthonormal 
columns. The second factor is called R, because the nonzeros are to the right of the di- 


agonal (and the letter U is already taken). The off-diagonal entries of R are the numbers 
qib = 1/\/2 and qic = gue = V2, found above. The whole factorization is 


Le 1/J/2 1/V2 0| |v2 1/v2 v2 
A=1|10 0 1/=|]| O 0 1 1//2 V2| =OR. 
100 1//2 -1/V2 0 1 


You see the lengths of a, B, C on the diagonal of R. The orthonormal vectors q1, 92, 43; 
which are the whole object of orthogonalization, are in the first factor Q. 

Maybe QR is not as beautiful as LU (because of the square roots). Both factoriza- 
tions are vitally important to the theory of linear algebra, and absolutely central to the 
calculations. If LU is Hertz, then OR is Avis. 
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The entries r;; = g}.a; appear in formula (11), when ||A ;||q; is substituted for A ;: 
a= (qiajq feee+ (q)-14))4j-1 + ||Aj\|gj = Q times column j of R. (13) 


3U Every m by n matrix with independent columns can be factored into 
A = QR. The columns of Q are orthonormal, and R is upper triangular and 
invertible. When m = n and all matrices are square, Q becomes an orthogonal 
matrix. 


I must not forget the main point of orthogonalization. It simplifies the least-squares 
problem Ax = b. The normal equations are still correct, but A'A becomes easier: 


A'A=R'Q'QOR=R'R. (14) 
The fundamental equation A'AX = A'D simplifies to a triangular system: 
R'RX=R'O'b or ~— RR =Q"D. (15) 
Instead of solving QRx = b, which can’t be done, we solve Rx = Q'b which is just 
back-substitution because R is triangular. The real cost is the mn” operations of Gram- 
Schmidt, which are needed to find Q and R in the first place. 

The same idea of orthogonality applies to functions, The sines and cosines are or- 
thogonal; the powers 1, x, x* are not. When f(x) is written as a combination of sines and 
cosines, that is a Fourier series. Each term is a projection onto a line—the line in func- 
tion space containing multiples of cosnx or sinnx. It is completely parallel to the vector 


case, and very important. And finally we have a job for Schmidt: To orthogonalize the 
powers of x and produce the Legendre polynomials. 


Function Spaces and Fourier Series 


This is a brief and optional section, but it has a number of good intentions: 
1. to introduce the most famous infinite-dimensional vector space (Hilbert space); 
2. to extend the ideas of length and inner product from vectors v to functions f(x): 


3. to recognize the Fourier series as a sum of one-dimensional projections (the orthog- 
onal “columns” are the sines and cosines); 


4. to apply Gram-Schmidt orthogonalization to the polynomials 1,x,x,...; and 
5. to find the best approximation to f(x) by a straight line. 
We will try to follow this outline, which opens up a range of new applications for 


linear algebra, in a systematic way. 


1. Hilbert Space. After studying R”, it is natural to think of the space R®. It con- 
tains all vectors v = (v1, v2,v3,...) with an infinite sequence of components. This space 
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is actually too big when there is no control on the size of components v;. A much better 
idea is to keep the familiar definition of length, using a sum of squares, and to include 
only those vectors that have a finite length: 


Length squared —__||v|/7 = v7 +v5+v34+--- (16) 


The infinite series must converge to a finite sum. This leaves (1, 7 7 ...) but not (1,1,1,...). 
Vectors with finite length can be added (||v-+w]| < ||v||+ ||w||) and multiplied by scalars, 
so they form a vector space. It is the celebrated Hilbert space. 

Hilbert space is the natural way to let the number of dimensions become infinite, 
and at the same time to keep the geometry of ordinary Euclidean space. Ellipses become 
infinite-dimensional ellipsoids, and perpendicular lines are recognized exactly as before. 
The vectors v and w are orthogonal when their inner product is zero: 


Orthogonality viw =viwy tows + v3w3+---=0. 


This sum is guaranteed to converge, and for any two vectors it still obeys the Schwarz 

inequality |v' w| < ||v||||w]|. The cosine, even in Hilbert space, is never larger than 1. 
There is another remarkable thing about this space: It is found under a great many 

different disguises. Its “vectors” can turn into functions, which is the second point. 


2. Lengths and Inner Products. Suppose f(x) = sinx on the interval 0 < x < 27. This 
f is like a vector with a whole continuum of components, the values of sinx along the 
whole interval. To find the length of such a vector, the usual rule of adding the squares 
of the components becomes impossible. This summation is replaced, in a natural and 
inevitable way, by integration: 


20 20 
Length || f|| of function —_|| f ||? = [ (f (x))?dx = [ (sinx)’dx=a (17) 


Our Hilbert space has become a function space. The vectors are functions, we have a 
way to measure their length, and the space contains all those functions that have a finite 
length—just as in equation (16). It does not contain the function F(x) = 1/x, because 
the integral of 1 /x? is infinite. 

The same idea of replacing summation by integration produces the inner product of 
two functions: If f(x) = sinx and g(x) = cosx, then their inner product is 


20 20 
(f.2) = fixjgla)dx= | sinxcosxdx = 0. (18) 
0 0 


This is exactly like the vector inner product f'g. It is still related to the length by 
(f, f) = ||f\|?. The Schwarz inequality is still satisfied: |(f,g)| < || f||||g||. Of course, 
two functions like sinx and cosx—whose inner product is zero—will be called orthogo- 
nal. They are even orthonormal after division by their length \/7. 
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3. The Fourier series of a function is an expansion into sines and cosines: 
f(x) =ao +a; cosx + by sinx + az cos2x + bz sin2x+---. 


To compute a coefficient like b;, multiply both sides by the corresponding function sinx 
and integrate from 0 to 27. (The function f(x) is given on that interval.) In other words, 
take the inner product of both sides with sinx: 


20 20 27 20 , 
f (x) sinxdx = ay | sinxdx-+ay | cosxsinxde+by / (sinx)“dx+---. 
0 0 0 0 


On the right-hand side, every integral is zero except one—the one in which sinx multi- 
plies itself. The sines and cosines are mutually orthogonal as in equation (18) Therefore 
b, is the left-hand side divided by that one nonzero integral: 


5, lo" fle)sinxdx __(f,sinx) 
a 5 (sinx)2dx ~ (sinx,sinx)’ 


The Fourier coefficient a; would have cos x in place of sinx, and az would use cos 2x. 

The whole point is to see the analogy with projections. The component of the vector 
b along the line spanned by ais b'a/a'‘a. A Fourier series is projecting f(x) onto sinx. 
Its component p in this direction is exactly b, sinx. 

The coefficient b; is the least squares solution of the inconsistent equation b, sinx = 
f(x). This brings b; sinx as close as possible to f(x). All the terms in the series are 
projections onto a sine or cosine. Since the sines and cosines are orthogonal, the Fourier 
series gives the coordinates of the “vector” f (x) with respect to a set of (infinitely many) 
perpendicular axes. 


4. Gram-Schmidt for Functions. There are plenty of useful functions other than 
sines and cosines, and they are not always orthogonal. The simplest are the powers of x, 
and unfortunately there is no interval on which even | and x* are perpendicular. (Their 
inner product is always positive, because it is the integral of x*.) Therefore the closest 
parabola to f(x) is not the sum of its projections onto 1, x, and x*. There will be a matrix 
like (ATA)~!, and this coupling is given by the ill-conditioned Hilbert matrix. On the 
interval O <x < 1, 


(1,1) (1.x) (1,x?) PL: Wee she 
ALAS |), Goals. Gee || oe fe foe = 
(sh Ga) Ge) fxr (ea fx 


WRN — 
Bl Rew ble 
Me BIR Go| 


This matrix has a large inverse, because the axes 1, x, x* are far from perpendicular. The 
situation becomes impossible if we add a few more axes. It is virtually hopeless to solve 
A'Ax = A'b for the closest polynomial of degree ten. 

More precisely, it is hopeless to solve this by Gaussian elimination; every roundoff 
error would be amplified by more than 10!°. On the other hand, we cannot just give 
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up; approximation by polynomials has to be possible. The right idea is to switch to 
orthogonal axes (by Gram-Schmidt). We look for combinations of 1, x, and x” that are 
orthogonal. 

It is convenient to work with a symmetrically placed interval like —1 <x < 1, because 
this makes all the odd powers of x orthogonal to all the even powers: 


1 1 
(1.2) = | xdx=0, el wdx =0. 
=A 4 


Therefore the Gram-Schmidt process can begin by accepting v; = | and vz = x as the 
first two perpendicular axes. Since (x,x*) = 0, it only has to correct the angle between 1 
and x*. By formula (10), the third orthogonal polynomial is 

Lx : | dx 1 

Orthogonalize V3=x7°- ae) 1- Cie: a = Ly ==, 

bd) (x,x) fe ldx 3 
The polynomials constructed in this way are called the Legendre polynomials and they 
are orthogonal to each other over the interval -1 <x < 1. 


1 | 1 ° xy]! 
207 = Pi. = |—— -— — 
Check (1.x :) I, (x ;) dx p * os 0. 


The closest polynomial of degree ten is now computable, without disaster, by projecting 
onto each of the first 10 (or 11) Legendre polynomials. 


5. Best Straight Line. Suppose we want to approximate y = x° by a straight line 
C+ Dx between x = 0 and x = 1. There are at least three ways of finding that line, and 
if you compare them the whole chapter might become clear! 


1. Solve [1 x] |[§] =2° by least squares. The equation ATAXY = A‘D is 


(1,1) (1,x)| fe] _ “2 ot Pee. i 
a 5 iy “4 ae Or a Ue 
(x, 1) (x,x) D ax ) 2 3 D 17 
2. Minimize E* = fo (x° —C — Dx)?dx = 7 —2C —4D+C* +CD+ 4D”. The deriva- 
tives with respect to C and D, after dividing by 2, bring back the soanil equations 
of method 1 (and the solution is C= —3,D D= 3): 
: $C aD 0 d : ete + ai 0. 
==, — — n = 
6 2 gn s 


3. Apply Gram-Schmidt to replace x by x — (1,x)/(1,1). That is x — oe which is or- 
thogonal to 1. Now the one-dimensional projections add to the best line: 


: cae eee i 4 1 
OD Gabe HPF (3) 


2 
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Problem Set 3.4 


1. 


(a) Write the four equations for fitting y = C + Dt to the data 
y=-4 at t=-2, y=-3 at t=-1 
y=-1 at t=1, y=0 at t=2. 
Show that the columns are orthogonal. 
(b) Find the optimal straight line, draw its graph, and write E?. 


(c) Interpret the zero error in terms of the original system of four equations in two 
unknowns: The right-hand side (—4, —3, —1,0) is in the space. 


. Project b = (0,3,0) onto each of the orthonormal vectors a, = (3, 5, —t) and az = 


(—4, 5, +), and then find its projection p onto the plane of a; and ap. 


. Find also the projection of b = (0,3,0) onto a3 = (5, —t, +), and add the three pro- 


jections. Why is P = ajat + anas + 303 equal to J? 


. If Q; and Q> are orthogonal matrices, so that Q'Q = I, show that Q1Q> is also 


orthogonal. If Q; is rotation through 0, and Q> is rotation through @, what is Q;Q2? 
Can you find the trigonometric identities for sin(@ + @) and cos(@ + @) in the matrix 
multiplication Q;Q2? 


. If wis a unit vector, show that Q = J — 2uu' is a symmetric orthogonal matrix. (It 


is a reflection, also known as a Householder transformation.) Compute Q when 
ji [5 alee Cee | 
eed) oe a 


. Find a third column so that the matrix 


1//3 1/14 
Q=|1/Vv3 2/v14 
1//3 -3/V14 


is orthogonal. It must be a unit vector that is orthogonal to the other columns; how 
much freedom does this leave? Verify that the rows automatically become orthonor- 
mal at the same time. 


. Show, by forming b'b directly, that Pythagoras’s law holds for any combination 


b =x\q1 +++: +Xndn of orthonormal vectors: ||b||? = x7 +---+.x2. In matrix terms, 
b = Ox, so this again proves that lengths are preserved: ||Qx||* = ||x||. 


. Project the vector b = (1,2) onto two vectors that are not orthogonal, a; = (1,0) 


and az = (1,1). Show that, unlike the orthogonal case, the sum of the two one- 
dimensional projections does not equal b. 


. Ifthe vectors q1, g2, g3 are orthonormal, what combination of g; and q2 is closest to 


q3? 


10. 


11. 
12. 


13. 


14. 


15. 


16. 


17. 


18. 


19. 
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If gq, and q2 are the outputs from Gram-Schmidt, what were the possible input vectors 
a and b? 
Show that an orthogonal matrix that is upper triangular must be diagonal. 


What multiple of aj = Ral should be subtracted from a2 = Fal to make the result 
orthogonal to a;? Factor a ‘al into OR with orthonormal vectors in Q. 


Apply the Gram-Schmidt process to 


and write the result in the form A = OR. 


From the nonorthogonal a, b, c, find orthonormal vectors q1, g2, 3: 


ay A 


Which fundamental subspace contains q3? What is the least-squares solution of 
Ax=bit b=|1 2°7|"? 


Express the Gram-Schmidt orthogonalization of a;, az as A = OR: 


Given n vectors a; with m components, what are the shapes of A, Q, and R? 


With the same matrix A as in Problem 16, and with b= [1 1 1]', use A = QR to 
solve the least-squares problem Ax = b. 


If A = QR, find a simple formula for the projection matrix P onto the column space 
of A. 


Show that these modified Gram-Schmidt steps produce the same C as in equation 
(10): 
C'=c—(qic)qi and = C=C*—(q3C*) qo. 


This is much more stable, to subtract the projections one at a time. 
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20. 


21. 


22. 


23. 


24. 


25s 
26. 
27. 


28. 


29. 


30. 
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In Hilbert space, find the length of the vector v = (1/V2,1/V4,1/\/8,...) and the 
length of the function f(x) = e* (over the interval 0 < x < 1). What is the inner 
product over this interval of e* and e-*? 


What is the closest function acosx + bsinx to the function f(x) = sin2x on the in- 
terval from —7 to 7? What is the closest straight line c + dx? 


By setting the derivative to zero, find the value of b; that minimizes 
27 5 
\|b1 sinx —cosx||? = | (b; sinx — cosx)“dx. 
0 


Compare with the Fourier coefficient b,. 


Find the Fourier coefficients ao, a;, b; of the step function y(x), which equals 1 on 
the interval 0 < x < wz and 0 on the remaining interval 7 < x < 27: 
(y, 1) (y,cos.x) b (y, sin) 


do = qq =o 
s (1,1) (cos.x, cos) (sin.x, sinx) 


Find the fourth Legendre polynomial. It is a cubic x° + ax” + bx+c that is orthogonal 


to 1, x, and x? — t over the interval —1 <x < 1. 


What is the closest straight line to the parabola y = x7 over —1 <x <1? 
In the Gram-Schmidt formula (10), verify that C is orthogonal to g; and qo. 


Find an orthonormal basis for the subspace spanned by a; = (1,—1,0,0), a2 = 
(O15 =170) ey (0/0), 


Apply Gram-Schmidt to (1,—1,0), (0,1,—1), and (1,0, —1), to find an orthonormal 
basis on the plane x; + .x2-+.x3 = 0. What is the dimension of this subspace, and how 
many nonzero vectors come out of Gram-Schmidt? 


(Recommended) Find orthogonal vectors A, B, C by Gram-Schmidt from a, b, c: 
a = (1,-1,0,0) b= (0,1,—1,0) c = (0,0,1,-1). 


A, B, C and a, b, c are bases for the vectors perpendicular to d = (1,1,1,1). 


If A = OR then A'A = R'R= triangular times triangular. Gram-Schmidt 
on A corresponds to elimination on A'A. Compare 
1 0O 0 
1 1 0 aay 
A= wih A'A=|]-1 2 -1 
0 -1 1 
0 -l1 2 
0 oO -1 


For A'A, the pivots are 2, 3, $ and the multipliers are —4 and — 5. 
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(a) Using those multipliers in A, show that column 1 of A and B = column 2 — 
5(column 1) and C = column 3 — $(column 2) are orthogonal. 


(b) Check that |/column 1 ||? = 2, ||B\|? = 3, and ||C||? = - using the pivots. 
31. True or false (give an example in either case): 


(a) Q~' is an orthogonal matrix when Q is an orthogonal matrix. 


(b) If Q (3 by 2) has orthonormal columns then ||Qx|| always equals ||x||. 


32. (a) Find a basis for the subspace S in R* spanned by all solutions of 
xy +x2+%3-—x4 = 0. 


(b) Find a basis for the orthogonal complement S~. 
(c) Find b; in S and bp in S* so that bj + by = b = (1,1, 1,1). 


3.5 The Fast Fourier Transform 


The Fourier series is linear algebra in infinite dimensions. The “vectors” are functions 
f(x); they are projected onto the sines and cosines; that produces the Fourier coefficients 
a; and b,. From this infinite sequence of sines and cosines, multiplied by a, and by, we 
can reconstruct f(x). That is the classical case, which Fourier dreamt about, but in actual 
calculations it is the discrete Fourier transform that we compute. Fourier still lives, but 
in finite dimensions. 

This is pure linear algebra, based on orthogonality. The input is a sequence of num- 
bers yo,.--,¥n—1, instead of a function f(x). The output co,...,C,—1 has the same length 
n. The relation between y and c is linear, so it must be given by a matrix. This is the 
Fourier matrix F , and the whole technology of digital signal processing depends on it. 
The Fourier matrix has remarkable properties. 

Signals are digitized, whether they come from speech or images or sonar or TV (or 
even oil exploration). The signals are transformed by the matrix F’, and later they can be 
transformed back—to reconstruct. What is crucially important is that F and F~! can be 
quick: 


F—! must be simple. The multiplications by F and F~' must be fast. 


Those are both true. F~! has been known for years, and it looks just like F. In fact, 
F is symmetric and orthogonal (apart from a factor \/n), and it has only one drawback: 
Its entries are complex numbers. That is a small price to pay, and we pay it below. The 
difficulties are minimized by the fact that all entries of F and F~' tare powers of a single 
number w. That number has w” = 1. 
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The 4 by 4 discrete Fourier transform uses w = i (and notice it = 1). The success of 
the whole DFT depends on F times its complex conjugate F: 


|! Poe > 2 1 1 

i ae), ta a aye (=i)3 

= Lge ed me me a ae o 
a et ae ean 


Immediately FF = 4/ tells us that F~! = F'/4. The columns of F are orthogonal (to give 
the zero entries in 47). The n by n matrices will have FF = nl. Then the inverse of F is 
just F /n. In a moment we will look at the complex number w = e27‘/" (which equals i 
forn = 4). 

It is remarkable that F' is so easy to invert. If that were all (and up to 1965 it was all), 
the discrete transform would have an important place. Now there is more. The multipli- 
cations by F and F~! can be done in an extremely fast and ingenious way. Instead of 
n~ separate multiplications, coming from the n7 entries in the matrix, the matrix-vector 
products Fc and F~!y require only snlogn steps. This rearrangement of the multiplica- 
tion is called the Fast Fourier Transform. 

The section begins with w and its properties, moves on to F~!, and ends with the 
FFT—the fast transform. The great application in signal processing is filtering, and the 
key to its success is the convolution rule. In matrix language, all “circulant matrices” 
are diagonalized by F’.. So they reduce to two FFTs and a diagonal matrix. 


Complex Roots of Unity 


Real equations can have complex solutions. The equation x? + 1 = 0 led to the invention 
of i (and also to —i!). That was declared to be a solution, and the case was closed. If 
someone asked about x* — i = 0, there was an answer: The square roots of a complex 
number are again complex numbers. You must allow combinations x + iy, with a real 
part x and an imaginary part y, but no further inventions are necessary. Every real or 
complex polynomial of degree n has a full set of n roots (possibly complex and possibly 
repeated). That is the fundamental theorem of algebra. 

We are interested in equations like x+ = 1. That has four solutions—the fourth roots 
of unity. The two square roots of unity are | and —1. The fourth roots are the square 
roots of the square roots, 1 and —1, i and —i. The number i will satisfy it = 1 because 
it satisfies 7 = —1. For the eighth roots of unity we need the square roots of i, and that 
brings us to w = (1 +i)//2. Squaring w produces (1+ 2i+i7)/2, which is i—because 
1+ is zero. Then w® = i* = 1. There has to be a system here. 

The complex numbers cos @ + isin @ in the Fourier matrix are extremely special. The 
real part is plotted on the x-axis and the imaginary part on the y-axis (Figure 3.11). Then 
the number w lies on the unit circle; its distance from the origin is cos* @ + sin? @ = 1. 
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It makes an angle @ with the horizontal. The whole plane enters in Chapter 5, where 
complex numbers will appear as eigenvalues (even of real matrices). Here we need only 
special points w, all of them on the unit circle, in order to solve w” = 1. 


= cos +2s1n 3 


Real axis 


Figure 3.11: The eight solutions to z® = 1 are 1,w,w?,...,w’ with w= (1+i)/vV2. 


The square of w can be found directly (it just doubles the angle): 
w” = (cos @ +isin@)* = cos” @ — sin? 6 + 2isin @ cos 0. 


The real part cos @ — sin? @ is cos2, and the imaginary part 2 sin@cos@ is sin26. 
(Note that i is not included; the imaginary part is a real number.) Thus w* = cos20 + 
isin20. The square of w is still on the unit circle, but at the double angle 20. That 
makes us suspect that w” lies at the angle n@, and we are right. 

There is a better way to take powers of w. The combination of cosine and sine is a 
complex exponential, with amplitude one and phase angle 0: 


cos @ +isin@ =e. (2) 


The rules for multiplying, like (e*)(e*) = e°, continue to hold when the exponents i0 are 
imaginary. The powers of w =e’ stay on the unit circle: 


1 22 
Powers of w wae wha ein = 18, (3) 


The nth power is at the angle 18. When n = —1, the reciprocal |/w has angle —9. If 
we multiply cos 8 + isin @ by cos(—@) + isin(—@), we get the answer 1: 


ee — (cos @ +isin @)(cos 0 —isin@) = cos’ @ + sin? @ = 1. 


Note. I remember the day when a letter came to MIT from a prisoner in New York, 
asking if Euler’s formula (2) was true. It is really astonishing that three of the key 
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functions of mathematics should come together in such a graceful way. Our best answer 
was to look at the power series for the exponential: 
-9\2 (793 
0 1129, i)" | (8) 
e =1+i0+ ii + 3) 
The real part 1 — 97/2 +--- is cos @. The imaginary part 9 — 6°/6-+--- is the sine, The 
formula is correct, and I wish we had sent a more beautiful proof. 


With this formula, we can solve w” = 1. It becomes e!”° = 1, so that n@ must carry 
us around the unit circle and back to the start. The solution is to choose 0 = 27/n: The 
“primitive” nth root of unity is 


20 


20 : 
= e2Mi/n — cog + isin. (4) 
n n 


Wn 
Its nth power is e?’, which equals 1. For n = 8, this root is (1 +i) /V2: 


: | eee mn ., mw I+i 
W4 = COS, Pisin. = and wg = cos — + isin — = —— 


2 4 4° 2 


The fourth root is at 8 = 90°, which is 4 (360°). The other fourth roots are the powers 
i? = —1, i? = —i, and i* = 1. The other eighth roots are the powers We, We, aed ,wé. The 
roots are equally spaced around the unit circle, at intervals of 27 /n. Note again that the 
square of wg is w4, which will be essential in the Fast Fourier Transform. The roots add 


up to zero. First 1+i—1—i=0, and then 
Sum of eighth roots 1+weg + we se +w = 0. (5) 


One proof is to multiply the left side by wg, which leaves it unchanged. (It yields wg + 
we aaa we and we equals |.) The eight points each move through 45°, but they remain 
the same eight points. Since zero is the only number that is unchanged when multiplied 
by wg, the sum must be zero. When n is even the roots cancel in pairs (like 1 + i = 0 
and i+ i> = 0). But the three cube roots of 1 also add to zero. 


The Fourier Matrix and Its Inverse 


In the continuous case, the Fourier series can reproduce f(x) over a whole interval. It 
uses infinitely many sines and cosines (or exponentials). In the discrete case, with only 
n coefficients cg,...,Cy—1 to choose, we only ask for equality at n points. That gives n 
equations. We reproduce the four values y = 2,4,6,8 when Fc = y: 


co + CFF + 2+ © 2 
co + ice + Poo + Pe3 = 4 
Fo=y 2 _ (6) 
co + fey + Po + PQ = 6 
co +t Pey + Peo + P7e3 8. 
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The input sequence is y = 2,4,6,8. The output sequence is co,c1,C¢2,c3. The four equa- 
tions (6) look for a four-term Fourier series that matches the inputs at four equally spaced 
points x on the interval from 0 to 27: 


Discrete ; cee /2 

: % 2ix Bix _ at ea 
pone Cotrcje + c2e" + 03@ 7 = 6 at x= 
Series 8 at x=32/2. 


Those are the four equations in system (6). At x = 27 the series returns yo = 2 and 
continues periodically. The Discrete Fourier Series is best written in this complex form, 
as a combination of exponentials e’ rather than sinkx and cos kx. 

For every n, the matrix connecting y to c can be inverted. It represents n equations, 
requiring the finite series cg + cje* +--+. (nterms) to agree with y (at n points). The first 
agreement is at x = 0, where co +--+ + Cp—1 = yo. The remaining points bring powers of 
w, and the full problem is Fc = y: 


1 1 1 1 co Yo 
1 ow w wr! Cl YI 
fea 1 w wt w2n-V) 6a |= | Sar |) (7) 


¢ 4p 42) (n—1)* 


W W W Cn-1 n—1 


There stands the Fourier matrix F with entries Fj, = w/*. Tt is natural to number the 
rows and columns from 0 to n — 1, instead of | to n. The first row has j = 0, the first 
column has k = 0, and all their entries are w? = 1. 

To find the c’s we have to invert F. In the 4 by 4 case, F~! was built from 1/i = —i. 


That is the general rule, that F~' comes from the complex number w—! = 7. It lies at 
the angle —27/n, where w was at the angle +27 /n: 
3V_ The inverse matrix is built from the powers of w—! = 1/w = W: 
1 1 1 1 
i) age We? wie) = 
-1 ~2 F 
F’=-|1 w 1 =—. (8) 
n n 
t nea ap 2 x ay GD? 
1 i 1 1 


1 
1 
Thus F=|1 27/3 4zi/3 has Fol = 
1 


eAti/3  8ni/3 


1 
1 en 2/3 9—4ni/3 
1 eW4ti/3 p—8ni/3 


Row j of F times column j of F~! is always (1 +1+---+1)/n=1. The harder part is 
off the diagonal, to show that row j of F times column k of F~! gives zero: 


1-1+wiw* 4 ww 4. pw Dip ODE if JFk. (9) 
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The key is to notice that those terms are the powers of W = w/w~*: 

1+W4+W?+--+W"!=0. (10) 
This number W is still a root of unity: W” = w/w" is equal to 1/1-* = 1. Since j 
is different from k, W is different from 1. It is one of the other roots on the unit circle. 
Those roots all satisfy 1+W +---+W"-! =0. Another proof comes from 


1-w"=(1-W)(1+W4+W7+---+Ww"!). (11) 


Since W” = 1, the left side is zero. But W is not 1, so the last factor must be zero. The 
columns of F are orthogonal. 


The Fast Fourier Transform 


Fourier analysis is a beautiful theory, and it is also very practical. To analyze a waveform 
into its frequencies is the best way to take a signal apart. The reverse process brings it 
back. For physical and mathematical reasons the exponentials are special, and we can 
pinpoint one central cause: If you differentiate e', or integrate it, or translate x to 
x+h, the result is still a multiple of e'*. Exponentials are exactly suited to differential 
equations, integral equations, and difference equations. Each frequency component goes 
its Own way, as an eigenvector, and then they recombine into the solution. The analysis 
and synthesis of signals—computing c from y and y from c—is a central part of scientific 
computing. 

We want to show that Fc and F~'y can be done quickly. The key is in the relation of 
F4 to Fy— or rather to two copies of F, which go into a matrix F,’: 


111 £1 1 1 
(Coe ae ae <1 
2 o— iscloseto FF; = 
‘11 2 i 6 @ 11 
Le 8 2 1 -1 
F4 contains the powers of w4 = i, the fourth root of 1. F;" contains the powers of w2 = —1, 


the square root of 1. Note especially that half the entries in F; are zero. The 2 by 2 
transform, done twice, requires only half as much work as a direct 4 by 4 transform. If 
64 by 64 transform could be replaced by two 32 by 32 transforms, the work would be 
cut in half (plus the cost of reassembling the results). What makes this true, and possible 
in practice, is the simple connection between we64 and w32: 


ee 
(wea)? = wo, or (emis) — 92Hi/32. 


The 32nd root is twice as far around the circle as the 64th root. If w+ = 1, then (w*)** = 
1. The mth root is the square of the nth root, if m is half of n: 

2 
n 


Ww. = Wm if m= 5n. (12) 
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The speed of the FFT, in the standard form presented here, depends on working with 
highly composite numbers like 2'° = 1024. Without the fast transform, it takes (1024)? 
multiplications to produce F times c (which we want to do often). By contrast, a fast 
transform can do each multiplication in only 5- 1024 steps. Itis 200 times faster, because 
it replaces one factor of 1024 by 5. In general it replaces n? multiplications by snl, 
when n is 2°. By connecting F, to two copies of F, /2, and then to four copies of F;,/4, 
and eventually to a very small F, the usual n” steps are reduced to snlog, n. 

We need to see how y = F,,c (a vector with n components) can be recovered from two 
vectors that are only half as long. The first step is to divide c itself, by separating its 
even-numbered components from its odd-numbered components: 


CSCC Ge) and CS (Cente: 


The coefficients just go alternately into c’ and c’. From those vectors, the half-size 
transform gives y’ = F,c’ and y” = F,,c". Those are the two multiplications by the 
smaller matrix F,. The central problem is to recover y from the half-size vectors y’ and 
y”, and Cooley and Tukey noticed how it could be done: 


3W The first m and the last m components of the vector y = F,,c are 
yj =y,+wly%, j=0,...,m—1 


(13) 
Yjtm = Yj — why}, j=0,...,m—1. 


Thus the three steps are: split c into c’ and c”, transform them by F,, into y’ 
and y”, and reconstruct y from equation (13). 


We verify in a moment that this gives the correct y. (You may prefer the flow graph 
to the algebra.) This idea can be repeated. We go from F024 to F5\7 to Fy56. The final 
count is snl, when starting with the power n = 2‘ and going all the way ton = 1—where 
no multiplication is needed. This number qnl satisfies the rule given above: twice the 
count for m, plus m extra multiplications, produces the count for n: 


1 1 


Another way to count: There are @ steps from n = 2‘ to n= 1. Each step needs n/2 
multiplications by D,,/2 in equation (13), which is really a factorization of F;,: 


F512 even-odd 


I D 
One FFT step Fi024 = | 512 512 


I5\2_ —Ds512 


| (14) 


F5,2| | permutation 


The cost is only slightly more than linear. Fourier analysis has been completely trans- 
formed by the FFT. To verify equation (13), split y; into even and odd: 
n=l m—1 
w=). wie, is identical to 
k=0 k 


m—1 
2kj (2k+1)j 
Wry Oe y Wn C2k+1 
=0 k=0 
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2 


< 18 Wm, the two sums are 


Each sum on the right has m = sn terms. Since w 
m—1 m—1 ; : 
j= Yi wiley + wh Do walce = yj + why. (15) 
k=0 k=0 
For the second part of equation (13), 7+ min place of j produces a sign change: 
k(j+1) kj 


Inside the sums, w,, remains w,; since wkm = ]kK=], 


Outside, wi” = —w/, because w = e27im/n — emi — _], 


The FFT idea is easily modified to allow other prime factors of n (not only powers of 2). 
If n itself is a prime, a completely different algorithm is used. 


Example 1. The steps from n = 4 to m = 2 are 


a) a) 
Fc 
Ci C2 
— — — |y 
C2 Cl 
Fyc" 
C3 C3 


Combined, the three steps multiply c by Fy to give y. Since each step is linear, it must 
come from a matrix, and the product of those matrices must be F4: 


Ai te 2 of 1 1 1 1 1 

1i2P?P 1 | 1 
Scalia | (16) 

1 rir i 1 —] 1 1 1 

17 ® ? 1 i f=] 1 


You recognize the two copies of F> in the center. At the right is the permutation matrix 


that separates c into c’ and c”. At the left is the matrix that multiplies by w). If we 


started with Fg, the middle matrix would contain two copies of Fy. Each of those would 
be split as above. Thus the FFT amounts to a giant factorization of the Fourier matrix! 
The single matrix F with n? nonzeros is a product of approximately @ = log, n matrices 
(and a permutation) with a total of only né nonzeros. 


The Complete FFT and the Butterfly 


The first step of the FFT changes multiplication by F, to two multiplications by Fin. 
The even-numbered components (cg, cz) are transformed separately from (c;,c3), Figure 
3.12 gives a flow graph for n = 4. For n = 8, the key idea is to replace each F, box by 
F> boxes. The new factor w4 = i is the square of the old factor w = wg = e2t/8 The 
flow graph shows the order that the c’s enter the FFT and the log, n stages that take them 
through it—and it also shows the simplicity of the logic. 

Every stage needs sn multiplications so the final count is snlog n. There is an amaz- 
ing rule for the overall permutation of c’s before entering the FFT: Write the subscripts 
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00 00 
10 01 
O1 10 
11 11 


Figure 3.12: Flow graph for the Fast Fourier Transform with n = 4. 


0,...,7 in binary and reverse the order of their bits. The subscripts appear in “bit- 
reversed order” on the left side of the graph. Even numbers come before odd (numbers 
ending in 0 come before numbers ending in 1). 


Problem Set 3.5 


1. What are F? and F* for the 4 by 4 Fourier matrix F? 


2. Find a permutation P of the columns of F that produces FP = F (n by n), Combine 
with FF = nl to find F? and F* for the n by n Fourier matrix. 


3. If you form a 3 by 3 submatrix of the 6 by 6 matrix F¢, keeping only the entries in 
its first, third, and fifth rows and columns, what is that submatrix? 


4. Mark all the sixth roots of 1 in the complex plane. What is the primitive root we? 
(Find its real and imaginary part.) Which power of w6 is equal to 1/w6? What is 
l+wtw+wt+witw? 


5. Find all solutions to the equation e = —1, and all solutions to e!9 = i. 


6. What are the square and the square root of wj28, the primitive 128th root of 1? 


7. Solve the 4 by 4 system (6) if the right-hand sides are yo = 2, y; = 0, yo = 2, y3 = 0. 
In other words, solve Fac = y. 


8. Solve the same system with y = (2,0,—2,0) by knowing F,' and computing c = 
‘re ly. Verify that cy + cje” + cre** + c3e** takes the values 2, 0, —2, 0 at the points 
x= 0,7 /2, 7,377) 2. 

9. (a) If y=(1,1,1,1), show that c = (1,0,0,0) satisfies Fyc = y. 

(b) Now suppose y = (1,0,0,0), and find c. 
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10. 


11. 


12. 


13. 


14. 


15. 
16. 
17. 
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For n = 2, write yo from the first line of equation (13) and y; from the second line. 
For n = 4, use the first line to find yo and y;, and the second to find y2 and y3, all in 
terms of y’ and y”. 


Compute y = Fic by the three steps of the Fast Fourier Transform if c = (1,0,1,0). 


Compute y = Fc by the three steps of the Fast Fourier Transform if c = (1,0, 1,0,1,0,1,0). 
Repeat the computation with c = (0,1,0,1,0,1,0,1). 


For the 4 by 4 matrix, write out the formulas for co, c1, c2, c3 and verify that if f is 
odd then c is odd. The vector f is odd if f,_; = —f;; for n = 4 that means fo = 0, 
fa =—/Si. fo = 0 as in sinO, sinz/2, sin, sin37/2. This is copied by c and it leads 
to a fast sine transform. 


Multiply the three matrices in equation (16) and compare with F’.. in which six entries 
do you need to know that i = —1? 


Invert the three factors in equation (14) to find a fast factorization of F~ e 
F is symmetric. So transpose equation (14) to find a new Fast Fourier Transform! 


All entries in the factorization of Fg involve powers of w = sixth root of 1: 


[?] 


Write these factors with 1, w, w* in D and 1, w”, w* in F;. Multiply! 


TDI Ie 
Fo = 


ae | ee 8 


Py 


Problems 18-20 introduce the idea of an eigenvector and eigenvalue, when a matrix 
times a vector is a multiple of that vector. This is the theme of Chapter 5. 


18. 


19. 


The columns of the Fourier matrix F are the eigenvectors of the cyclic permutation 
P. Multiply PF to find the eigenvalues Ag to A3: 


Ont 00) (4: We ae a Cae TAG 
0010/]1 i? BP) Jt i? Pe Ay 
0:0. OL) Mae eae ae ee Ar 
1 O° 0: OP i. eo 2 Cee cas aa A3 


This is PF = FA or P= FAF™!. 


Two eigenvectors of this circulant matrix C are (1,1,1,1) and (1,7,i7,i°). What are 
the eigenvalues eg and e;? 


co ci 6c2) 3] Jl 1 1 1 
C3 Co. OCy.COC 1 i i 
= €0 and C 2) = el 2 
C2 C3 COCI 1 1 i i 
C13. ~C%:. cgi) | 1 1 i> i 


20. 


21. 


22. 
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Find the eigenvalues of the “periodic” —1, 2, —1 matrix C. The —1s in the corners 
of C make it periodic (a circulant matrix): 


C= has co= 2, cy = —1, co = 0, c3 = -1. 


To multiply C times x, when C= FEF ~!, we can multiply F(E(F~!x)) instead. The 
direct Cx uses n? separate multiplications. Knowing E and F, the second way uses 
only nlog,n-+n multiplications. How many of those come from EF, how many from 
F, and how many from F~!? 


How could you quickly compute these four components of Fc starting from co + c2, 
Co — C2, C1 +3, C1 — C3? You are finding the Fast Fourier Transform! 


cote, +c2+ 3 
coticy +P er +iPc3 
cote, titer 4+ fc3 
cote, +i®oo +3 


Fco= 


Review Exercises 


3.1 


3.2 


3.3 
3.4 
3.5 
3.6 


3.7 


3.8 


3.9 


Find the length of a = (2,—2,1), and write two independent vectors that are per- 
pendicular to a. 


Find all vectors that are perpendicular to (1,3,1) and (2,7,2), by making those the 
rows of A and solving Ax = 0. 


What is the angle between a = (2, —2,1) and b = (1,2,2)? 
What is the projection p of b = (1,2,2) onto a = (2,—2,1)? 
Find the cosine of the angle between the vectors (3,4) and (4,3), 


Where is the projection of b = (1,1,1) onto the plane spanned by (1,0,0) and 
(1,1,0)? 


The system Ax = b has a solution if and only if b is orthogonal to which of the four 
fundamental subspaces? 


Which straight line gives the best fit to the following data: b= 0 att =0,b=O at 
t=1,b=12att=3? 


Construct the projection matrix P onto the space spanned by (1, 1,1) and (0, 1,3). 
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3.10 


3.11 
3.12 
3.13 


3.14 


3.15 
3.16 


3.17 


3.18 


3.19 


3.20 


3.21 


3.22 


3.23 
3.24 


3.25 
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Which constant function is closest to y = x* (in the least-squares sense) over the 
intervalO<x< 1? 


If Q is orthogonal, is the same true of Q?? 
Find all 3 by 3 orthogonal matrices whose entries are zeros and ones. 


What multiple of a; should be subtracted from a2, to make the result orthogonal to 
a,? Sketch a figure. 


Factor 
cos@ sin®@ 
sn@ 0 
into QR, recognizing that the first column is already a unit vector. 
If every entry in an orthogonal matrix is either i or — i how big is the matrix? 


Suppose the vectors g1,...,gn are orthonormal. If b = cig, +---+ Cnn, give a 
formula for the first coefficient c; in terms of b and the q’s. 


What words describe the equation A'Ax = A'D, the vector p = Ax = Pb, and the 
matrix P = A(ATA)~!A?? 


If the orthonormal vectors g, = (3, 5, —4) and q2 = (—4, 5, +) are the columns of 
Q, what are the matrices Q'Q and QQ"? Show that QQ! is a projection matrix 
(onto the plane of qg; and q2). 


If vj,...,¥, is an orthonormal basis for R”, show that vjv! +---+vyvl =1. 
1 n 


True or false: If the vectors x and y are orthogonal, and P is a projection, then Px 
and Py are orthogonal. 


Try to fit a line b = C+ Dt through the points b = 0, t = 2, and b = 6, t = 2, and show 
that the normal equations break down. Sketch all the optimal lines, minimizing the 
sum of squares of the two errors. 


What point on the plane x + y — z = 0 is closest to b = (2,1,0)? 
Find an orthonormal basis for R? starting with the vector (1,1, 1). 


CT scanners examine the patient from different directions and produce a matrix 
giving the densities of bone and tissue at each point. Mathematically, the problem 
is to recover a matrix from its projections. in the 2 by 2 case, can you recover the 
matrix A if you know the sum along each row and down each column? 


Can you recover a 3 by 3 matrix if you know its row sums and column sums, and 
also the sums down the main diagonal and the four other parallel diagonals? 


3.26 


3.27 


3.28 


3.29 


3.30 


3.31 


3.32 


3.33 
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Find an orthonormal basis for the plane x — y+ z= 0, and find the matrix P that 
projects onto the plane. What is the nullspace of P? 


Let A = [3 1 1], and let V be the nullspace of A. 


(a) Find a basis for V and a basis for V+. 


(b) Write an orthonormal basis for V+, and find the projection matrix P, that projects 
vectors in R? onto V-. 


(c) Find the projection matrix P) that projects vectors in R? onto V. 
Use Gram-Schmidt to construct an orthonormal pair gi, g2 from a; = (4,5,2,2) 


and a2 = (1,2,0,0), Express a; and az as combinations of g; and q2, and find the 
triangular R in A = QR. 


For any A, b, x, and y, show that 


(a) if Ax =b and y'A =0, then y'b =0. 
(b) if Ax = 0 and Aly = 5, then x'b =0. 


What theorem does this prove about the fundamental subspaces? 


Is there a matrix whose row space contains (1,1,0) and whose nullspace contains 
(0,1,1)? 


The distance from a plane a'x = c (in m-dimensional space) to the origin is |c| /||al|. 


How far is the plane x; + x2 — x3 — x4 = 8 from the origin, and what point on it is 
nearest? 


In the parallelogram with corners at 0, v, w, and v+ w, show that the sum of the 
squared lengths of the four sides equals the sum of the squared lengths of the two 
diagonals. 


(a) Find an orthonormal basis for the column space of A. 


1 —6 
3 6 
A=|4 8 
5 0 
7 8 


(b) Write A as QR, where Q has orthonormal columns and R is upper triangular. 


(c) Find the least-squares solution to Ax = b, if b = (—3,7,1,0,4). 


3.34 With weighting matrix W = [+ 4], what is the W-inner product of (1,0) with (0, 1)? 
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3.35 


3.36 


3.37 


3.38 


3.39 
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To solve a rectangular system Ax = b, we replace A~! (which doesn’t exist) by 
(ATA)~!A? (which exists if A has independent columns). Show that this is a left- 
inverse of A but not a right-inverse. On the left of A it gives the identity; on the right 
it gives the projection P. 


Find the straight line C+ Dt that best fits the measurements b = 0,1,2,5 at times 
t =0,1,3,4. 


Find the curve y = C+ D2’ which gives the best least-squares fit to the measure- 
ments y= 6 att=0, y=4att=1,y=0O att =2. Write the three equations that 
are solved if the curve goes through the three points, and find the best C and D. 


If the columns of A are orthogonal to each other what can you say about the form 
of AA? If the columns are orthonormal, what can you say then? 


Under what condition on the columns of A (which may be rectangular) is ATA in- 
vertible? 


Determinants 


4.1. Introduction 


Determinants are much further from the center of linear algebra than they were a hundred 
years ago. Mathematics keeps changing direction! After all, a single number can tell 
only so much about a matrix. Still, it is amazing how much this number can do. 

One viewpoint is this: The determinant provides an explicit “formula” for each entry 
of A~! and A~'b. This formula will not change the way we compute; even the deter- 
minant itself is found by elimination. In fact, elimination can be regarded as the most 
efficient way to substitute the entries of an n by n matrix into the formula. What the 
formula does is to show how A~! depends on the n? entries of A, and how it varies when 
those entries vary. 

We can list four of the main uses of determinants: 


1. They test for invertibility. If the determinant of A is zero, then A is singular. If 
detA + 0, then A is invertible (and A~! involves 1/detA). 

The most important application, and the reason this chapter is essential to the book, 
is to the family of matrices A— AJ. The parameter A is subtracted all along the main 
diagonal, and the problem is to find the eigenvalues for which A — AJ is singular. The 
test is det(A — AJ) = 0. This polynomial of degree n in A has exactly n roots. The matrix 
has n eigenvalues, This is a fact that follows from the determinant formula, and not from 
a computer. 


2. The determinant of A equals the volume of a box in n-dimensional space. The edges 
of the box come from the rows of A (Figure 4.1). The columns of A would give an 
entirely different box with the same volume. 

The simplest box is a little cube dV = dxdydz, as in J{f f(x,y,z)dV. Suppose we 
change to cylindrical coordinates by x = rcos 8, y=rsin@, z= z. Just as a small inter- 
val dx is stretched to (dx/du)du—when u replaces x in a single integral—so the volume 
element becomes J drd@dz. The Jacobian determinant is the three-dimensional ana- 
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(a31, 432, 433) 


(a21, a22, a23) 


(a11, 412, 413) 


Xx 


Figure 4.1: The box formed from the rows of A: volume = |determinantl. 


logue of the stretching factor dx/du: 


Ox/dr dx/08 dx/dz| |cos@ —rsind 0 
Jacobian J={|dy/dr dy/0@ dy/dz|=|sin@ rcos@ Oj. 
Az/dr dz/d0 dz/dz 0 0 1 


The value of this determinant is J = r. It is the r in the cylindrical volume element 
r drd@ dz; this element is our little box. (It looks curved if we try to draw it, but proba- 
bly it gets straighter as the edges become infinitesimal.) 


3. The determinant gives a formula for each pivot. Theoretically, we could predict 
when a pivot entry will be zero, requiring a row exchange. From the formula determi- 
nant = + (product of the pivots), it follows that regardless of the order of elimination, 
the product of the pivots remains the same apart from sign. 

Years ago, this led to the belief that it was useless to escape a very small pivot by 
exchanging rows, since eventually the small pivot would catch up with us. But what 
usually happens in practice, if an abnormally small pivot is not avoided, is that it is very 
soon followed by an abnormally large one. This brings the product back to normal but 
it leaves the numerical solution in ruins. 


4. The determinant measures the dependence of A~'b on each element of b. If one 
parameter is changed in an experiment, or one observation is corrected, the “influence 
coefficient” in A~! is a ratio of determinants. 

There is one more problem about the determinant. It is difficult not only to decide 
on its importance, and its proper place in the theory of linear algebra, but also to choose 
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the best definition. Obviously, detA will not be some extremely simple function of n* 
variables; otherwise A~! would be much easier to find than it actually is. 


The simple things about the determinant are not the explicit formulas, but the prop- 
erties it possesses. This suggests the natural place to begin. The determinant can be (and 
will be) defined by its three most basic properties: det/ = 1, the sign is reversed by a 
row exchange, the determinant is linear in each row separately. The problem is then 
to show, by systematically using these properties, how the determinant can be computed. 
This will bring us back to the product of the pivots. 

Section 4.2 explains these three defining properties of the determinant, and their most 
important consequences. Section 4.3 gives two more formulas for the determinant—the 
“big formula” with n! terms, and a formula “by induction”. In Section 4.4 the determi- 
nant is applied to find A~!. Then we compute x = A~'b by Cramer’s rule. And finally, in 
an optional remark on permutations, we show that whatever the order in which the prop- 
erties are used, the result is always the same—the defining properties are self-consistent. 

Here is a light-hearted question about permutations. How many exchanges does it 
take to change VISA into AVIS? Is this permutation odd or even? 


4.2 Properties of the Determinant 


This will be a pretty long list. Fortunately each rule is easy to understand, and even 
easier to illustrate, for a 2 by 2 example. Therefore we shall verify that the familiar 
definition in the 2 by 2 case, 


det E ; — 
cd 


possesses every property in the list. (Notice the two accepted notations for the deter- 
minant, detA and |A|.) Properties 4-10 will be deduced from the previous ones. Every 
property is a consequence of the first three. We emphasize that the rules apply to 
square matrices of any size. 


a b 


= ad -— be, 
Ca 


1. The determinant of the identity matrix is 1. 


1 0 


dares 
01 


1 0 
=| and 0 1 
0 0 
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2. The determinant changes sign when two rows are exchanged. 


omes a b 
Row exchange = cb—ad = — 
a b eid 
The determinant of every permutation matrix is detP = +1. By row exchanges, we can 


turn P into the identity matrix. Each row exchange switches the sign of the determinant, 
until we reach detJ = 1. Now come all other matrices! 


3. The determinant depends linearly on the first row. Suppose A, B, C are the same 
from the second row down—and row | of A is a linear combination of the first rows of 
B and C. Then the rule says: detA is the same combination of detB and detC. 

Linear combinations involve two operations—adding vectors and multiplying by scalars. 
Therefore this rule can be split into two parts: 


/ / / b! 
Add vectors in row 1 Bae OTe zai: + ie . 
Cc d Ca c d 
tb b 
Multiply by ¢ in row 1 ie = ‘ er 


Notice that the first part is not the false statement det(B+C) = detB+detC. You cannot 
add all the rows: only one row is allowed to change. Both sides give the answer ad + 
a'd—bc—b'c. 

The second part is not the false statement det(tA) = tdetA. The matrix tA has a factor 
t in every row (and the determinant is multiplied by ¢”). It is like the volume of a box, 
when all sides are stretched by 4. In n dimensions the volume and determinant go up by 
4”. If only one side is stretched, the volume and determinant go up by 4; that is rule 3. 
By rule 2, there is nothing special about the first row. 

The determinant is now settled, but that fact is not at all obvious. Therefore we grad- 
ually use these rules to find the determinant of any matrix. 


4. If two rows of A are equal, then detA = 0. 


ab 
ab 


Equal rows =ab—ba=0. 


This follows from rule 2, since if the equal rows are exchanged, the determinant is sup- 
posed to change sign. But it also has to stay the same, because the matrix stays the same. 
The only number which can do that is zero, so detA = 0. (The reasoning fails if 1 = —1, 
which is the case in Boolean algebra. Then rule 4 should replace rule 2 as one of the 
defining properties.) 
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5. Subtracting a multiple of one row from another row leaves the same determinant. 


Row operation 


Rule 3 would say that there is a further term —? | ¢ 4, but that term is zero by rule 4. The 
usual elimination steps do not affect the determinant! 


6. If A has a row of zeros, then detA = 0. 


0 0 
c d 


Zero row = 0. 


One proof is to add some other row to the zero row. The determinant is unchanged, by 
rule 5. Because the matrix will now have two identical rows, detA = 0 by rule 4. 


7. If A is triangular then detA is the product a,,d22-++Ayyn of the diagonal entries. If 
the triangular A has Is along the diagonal, then detA = 1. 


a b 
0d 


a 0 


= 0d 
cd 


Triangular matrix = ad. 


Proof. Suppose the diagonal entries are nonzero. Then elimination can remove all the 
off-diagonal entries, without changing the determinant (by rule 5). If A is lower triangu- 
lar, the steps are downward as usual. If A is upper triangular, the /ast column is cleared 
out first—using multiples of a,,. Either way we reach the diagonal matrix D: 


aii 
D= Nik has detD = aj1d22--- Any, detl = aj1A22°-+ Any. 
Ann 


To find detD we patiently apply rule 3. Factoring out a;; and then a2 and finally ayy 
leaves the identity matrix. At last we have a use for rule 1: det/ = 1. 


If a diagonal entry is zero then elimination will produce a zero row. By rule 5 these 
elimination steps do not change the determinant. By rule 6 the zero row means a zero 
determinant. This means: When a triangular matrix is singular (because of a zero on the 
main diagonal) its determinant is zero. 

This is a key property. All singular matrices have a zero determinant. 


8. If A is singular, then detA = 0. If A is invertible, then detA # 0. 


Singular matrix j : is not invertible if and only if ad—bc =0. 
c 
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If A is singular, elimination leads to a zero row in U. Then detA = detU =0. If A 
is nonsingular, elimination puts the pivots d;,...,d, on the main diagonal. We have a 
“product of pivots” formula for detA! The sign depends on whether the number of row 


exchanges is even or odd: 


Product of pivots detA = +detU = +d d2---dy. 
The ninth property is the product rule. I would say it is the most surprising. 


9. The determinant of AB is the product of detA times det B. 


ae 
gh 


ab 
cd 


ae+bg af+bh 
ce+dg cf+dh| 


Product rule |A||B| = |AB| 


A particular case of this rule gives the determinant of A~!. It must be 1/detA: 


1 

detA~' = ——_ because (detA)(detA~!) = detAA~! = det/ = 1. 
detA 

In the 2 by 2 case, the product rule could be patiently checked: 


(ad — bc)(eh — fg) = (ae+bg)(cf +dh) — (af + bh)(ce+dg). 


(1) 


(2) 


In the n by n case we suggest two possible proofs—since this is the least obvious rule. 
Both proofs assume that A and B are nonsingular; otherwise AB is singular, and the 


equation detAB = (detA) (det B) is easily verified. By rule 8, it becomes 0 = 0. 


(i) We prove that the ratio d(A) = detAB/detB has properties 1-3. Then d(A) must 
equal detA. For example, d(/) = detB/detB = 1; rule 1 is satisfied. If two rows 
of A are exchanged, so are the same two rows of AB, and the sign of d changes 
as required by rule 2. A linear combination in the first row of A gives the same 
linear combination in the first row of AB. Then rule 3 for the determinant of AB, 
divided by the fixed quantity det B, leads to rule 3 for the ratio d(A). Thus d(A) = 


detAB/ det B coincides with detA, which is our product formula. 


(ii) This second proof is less elegant. For a diagonal matrix, det DB = (det D) (det B) 
follows by factoring each d; from its row. Reduce a general matrix A to D by 
elimination—from A to U as usual, and from U to D by upward elimination. The 
determinant does not change, except for a sign reversal when rows are exchanged. 
The same steps reduce AB to DB, with precisely the same effect on the determinant. 


But for DB it is already confirmed that rule 9 is correct. 


10. The transpose of A has the same determinant as A itself: detA' = detA. 


a b 
cd 


ac 
bd 


Transpose rule 4] = = a ja | , 
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Again the singular case is separate; A is singular if and only if A! is singular, and we 
have 0 = 0. If A is nonsingular, then it allows the factorization PA = LDU, and we apply 
rule 9 for the determinant of a product: 


det PdetA = detLdetDdetU. (3) 
Transposing PA = LDU gives AP! = U'D'L", and again by rule 9, 
detA’ det P’ = detU' detD' detL". (4) 


This is simpler than it looks, because L, U, L', and U? are triangular with unit diagonal. 
By rule 7, their determinants all equal 1. Also, any diagonal matrix is the same as its 
transpose: D = D!. We only have to show that det P = det P'. 

Certainly det P is 1 or —1, because P comes from J by row exchanges. Observe also 
that PP™ = J. (The 1 in the first row of P matches the 1 in the first column of P?, and 
misses the 1s in the other columns.) Therefore det Pdet P! = det/ = 1, and P and P! 
must have the same determinant: both | or both —1. 

We conclude that the products (3) and (4) are the same, and detA = detA!. This 
fact practically doubles our list of properties, because every rule that applied to the rows 
can now be applied to the columns: The determinant changes sign when two columns 
are exchanged, two equal columns (or a column of zeros) produce a zero determinant, 
and the determinant depends linearly on each individual column. The proof is just to 
transpose the matrix and work with the rows. 

I think it is time to stop and call the list complete. It only remains to find a definite 
formula for the determinant, and to put that formula to use. 


Problem Set 4.2 

1. Ifa 4 by 4 matrix has detA = 5, find det(2A), det(—A), det(A”), and det(A~'). 

2. Ifa3 by 3 matrix has detA = —1, find det(5A), det(—A), det(A”), and det(A~!). 

3. Row exchange: Add row 1 of A to row 2, then subtract row 2 from row 1. Then 
add row | to row 2 and multiply row | by —1 to reach B. Which rules show the 


following? 


det B = |“ ‘ equals —detA = — 
a b 


Those rules could replace Rule 2 in the definition of the determinant. 
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4. By applying row operations to produce an upper triangular U, compute 


fe 0. 2G 2) 0. 6 
2 = = = 

det 2 ae and det Bee re 
—-1 —2 0 2 Oo -1 2 -!1 
0 2 5 3 0 QO —-l —2 


Exchange rows 3 and 4 of the second matrix and recompute the pivots and determi- 
nant. 


Note. Some readers will already know a formula for 3 by 3 determinants. It has six 
terms (equation (2) of the next section), three going parallel to the main diagonal and 
three others going the opposite way with minus signs. There is a similar formula for 
4 by 4 determinants, but it contains 4! = 24 terms (not just eight). You cannot even 
be sure that a minus sign goes with the reverse diagonal, as the next exercises show. 


5. Count row exchanges to find these determinants: 


000 0100 
det eee taal | and det Sead =-—l. 
0100 0001 
1000 1000 
6. For each n, how many exchanges will put (row n, row n—1,..., row 1) into the 
normal order (row 1, ... , row n—1, row n)? Find detP for the n by n permutation 


with 1s on the reverse diagonal. Problem 5 had n = 4. 
7. Find the determinants of: 


(a) arank one matrix 


1 
A=|4 2 =i 2]. 
2 


(b) the upper triangular matrix 


oN NO 
YAN w 


(c) the lower triangular matrix U aa 


(d) the inverse matrix U~!. 


10. 


11. 


12. 


13. 


14. 
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(e) the “reverse-triangular’ matrix that results from row exchanges, 


03 0.:0° 2 
ya |o 026 
O22 
448 8 


Show how rule 6 (det = 0 if a row is zero) comes directly from rules 2 and 3. 


Suppose you do two row operations at once, going from 


a b ‘is a—mc b—md 
Cd c—fla d—tb|- 
Find the determinant of the new matrix, by rule 3 or by direct calculation. 


If Q is an orthogonal matrix, so that Q'Q = J, prove that detQ equals +1 or —1. 
What kind of box is formed from the rows (or columns) of Q? 


Prove again that detQ = | or —1 using only the Product rule. If |detQ| > 1 then 
det Q” blows up. How do you know this can’t happen to Q”? 


Use row operations to verify that the 3 by 3 “Vandermonde determinant” is 


laa 
det|1 b b*| =(b—a)(c—a)(c—b). 
lec e 
(a) A skew-symmetric matrix satisfies K' = —K, asin 
0 a b 
K={|-a 0 ¢ 
—b -c 0 


In the 3 by 3 case, why is det(—K) = (—1)*detK? On the other hand det K? = 
det K (always). Deduce that the determinant must be zero. 


(b) Write down a 4 by 4 skew-symmetric matrix with det K not zero. 
True or false, with reason if true and counterexample if false: 


(a) If A and B are identical except that b;; = 2a1,, then detB = 2detA. 
(b) The determinant is the product of the pivots. 

(c) If A is invertible and B is singular, then A + B is invertible. 

(d) If A is invertible and B is singular, then AB is singular. 

(e) The determinant of AB — BA is zero. 
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15. 


16. 


17. 


18. 


19. 


20. 


21. 
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If every row of A adds to zero, prove that detA = 0. If every row adds to 1, prove 
that det(A — 7) = 0. Show by example that this does not imply detA = 1. 


Find these 4 by 4 determinants by Gaussian elimination: 


11 12 13 14 fi cge gr 3 
24° 22> 23° 9A 1 
det : and det ; 
31 32 33 34 tz ¢ 
41 42 43 44 rt 


Find the determinants of 


4 2 . pete 3. -—2 | A-dI= 
1 3 10;-1 4 


For which values of A is A— AJ a singular matrix? 


A= 


Evaluate detA by reducing the matrix to triangular form (rules 5 and 7). 


hk. 33 Loi 3 1 
A=1]0 4 6], B= |0 4 6], C= 10 
15 8 0 0 1 1 


What are the determinants of B, C, AB, A'A, and C!? 


Suppose that CD = —DC, and find the flaw in the following argument: Taking de- 
terminants gives (detC)(detD) = —(detD)(detC), so either detC = 0 or detD = 0. 


Thus CD = —DC is only possible if C or D is singular. 
Do these matrices have determinant 0, 1, 2, or 3? 


00 1 0 
A= ]|1 0 0 B=}1 
010 1 


The inverse of a 2 by 2 matrix seems to have determinant = 1: 


— — 


49, 


a ee 


detA~! = det 


1 d —b (GAS De" 
ad — be ~~ ad—be | 


—c a 
What is wrong with this calculation? What is the correct detA~!? 
Problems 22-28 use the rules to compute specific determinants. 


. Reduce A to U and find detA = product of the pivots: 


23. 


24. 


25. 


26. 
27. 
28. 


29. 


30. 


31. 


32. 
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By applying row operations to produce an upper triangular U, compute 


Ly 2-3°0 214141 
2 

det oO 4 and det ee Sh 
—-1 0 0 3 | a es eae | 
0 207 Lr by eZ 


Use row operations to simplify and compute these determinants: 


101 201 301 i See a 4 
det |}102 202 302 and det|+ 1 tf 
103 203 303 Pt 
Elimination reduces A to U. Then A = LU: 
3. 3 4 1 O O}] 13 3 4 
A=|6 8 7/,=[/2 1 O| 10 2 —-1] =LU. 
—3 5 -9 —1 4 1/10 0 -1 


Find the determinants of L, U, A, U~'L~!, and U~!L~!A. 
If aj; is i times j, show that detA = 0. (Exception when A = [1].) 
If ajj is i+ j, show that detA = 0. (Exception when n = 1 or 2.) 
Compute the determinants of these matrices by row operations: 
0 a 0 aaa 
A=|0 0 Dl, B= 5g fe amd C= la bb 
c 0 0 000 abe 


What is wrong with this proof that projection matrices have det P = 1? 


1 


P=A(A'A)1A™ so |P|=|A/——— 
|AT||A] 


|A7| = 1. 


(Calculus question) Show that the partial derivatives of In(detA) give A7!: 


df/da a Hs jek 


f(a,b,¢,d) =In(ad— be) _ leads to be df /ad 


(MATLAB) The Hilbert matrix hilb(n) has i, j entry equal to 1/(i+ j—1). Print 
ti determinants of hilb(1), hilb(2),...,hilb(10). Hilbert matrices are hard to work 
with! What are the pivots? 


(MATLAB) What is a typical determinant (experimentally) of rand(n) and randn(n) 
for n = 50, 100,200,400? (And what does “Inf” mean in MATLAB?) 
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33. Using MATLAB, find the largest determinant of a 4 by 4 matrix of Os and 1s. 


34. If you know that detA = 6, what is the determinant of B? 


row 1 row 1+ row 2 
detA = |row 2| = 6 det B = [row 2+ row 3] = 
row 3 row 3+ row 1 


35. Suppose the 4 by 4 matrix M has four equal rows all containing a, b, c, d. We know 
that det(M) = 0. The problem is to find det(J + M) by any method: 


l+a b C d 
a 1+b c¢ d 

bo. ASE vd 
a b c l+d 


det(/+M) = 


Partial credit if you find this determinant when a = b =c =d = 1. Sudden death if 
you say that det(J + M) = det/ + detM. 


4.3. Formulas for the Determinant 


The first formula has already appeared. Row operations produce the pivots in D: 


4A If A is invertible, then PA = LDU and det P = +1. The product rule gives 


detA = +detLdetDdetU = +(product of the pivots). (1) 


The sign +1 depends on whether the number of row exchanges is even or odd. 
The triangular factors have detL = detU = 1 and detD = d,--+dy. 


In the 2 by 2 case, the standard LDU factorization is 


a bj) |1 O|}Ja 0 1 b/a 
c d| |c/a 1] |0 (ad—be)/a} |O 1 |’ 


The product of the pivots is ad — bc. That is the determinant of the diagonal matrix D. 
If the first step is a row exchange, the pivots are c and (— detA)/c. 


Example 1. The —1,2,—1 second difference matrix has pivots 2/1,3/2,... in D: 


2 = 2 


| 7 5 | (n+1)/n 
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Its determinant is the product of its pivots. The numbers 2,...,7 all cancel: 


o-2(3)() CE) 


MATLAB computes the determinant from the pivots. But concentrating all information 
into the pivots makes it impossible to figure out how a change in one entry would affect 
the determinant. We want to find an explicit expression for the determinant in terms of 
the n* entries. 


For n = 2, we will be proving that ad — bc is correct. For n = 3, the determinant 
formula is again pretty well known (it has six terms): 


aj, Qj2 a43 
+41 1022433 + @12423a31 + 413d21a32 

a2) a22. a23) = (2) 
—€@1 123032 — A12A21A33 — a13a2243}]. 

a3] 432 433 


Our goal is to derive these formulas directly from the defining properties 1-3 of detA. If 
we can handle n = 2 and n = 3 in an organized way, you will see the pattern. 


To start, each row can be broken down into vectors in the coordinate directions: 
la b) = |a 0) + [o b| and E d| = |e 0) + [o dj. 


Then we apply the property of linearity, first in row | and then in row 2: 


a bj _|ja 0 0 b 
Separate into Sal Versa Nhe. 
n” = 2° easy (3) 
determinants a 0| la O| |0 bl 10 b 
~ le ol lo dl |e o}' jo af 


Every row splits into n coordinate directions, so this expansion has n” terms. Most of 
those terms (all but n! = n factorial) will be automatically zero. When two rows are in 
the same coordinate direction, one will be a multiple of the other, and 


a 0 
c 0 


0 b 
Od 


= 0, =. 


We pay attention only when the rows point in different directions. The nonzero terms 
have to come in different columns. Suppose the first row has a nonzero term in column 
a, the second row is nonzero in column f, and finally the nth row in column v. The 
column numbers a, B,...,v are all different. They are a reordering, or permutation, of 
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the numbers 1,2,...,n. The 3 by 3 case produces 3! = 6 determinants: 


ai, @12 443 ait a2 a3 
a2) 422. a23| = a22 - a73| + |a21 
Q31 432 433 433; |@31 a32 (4) 
ait a2 a}3 
+ a23| + |a21 + an 
a32 33 a3] 


All but these n! determinants are zero, because a column is repeated. (There are 
n choices for the first column a, n — | remaining choices for B, and finally only one 
choice for the last column v. All but one column will be used by that time, when we 
“snake” down the rows of the matrix). In other words, there are n! ways to permute the 
numbers 1,2,...,n. The column numbers give the permutations: 


Column numbers (a, 8,v)=(1,2,3), (2,3,1), (3,1,2), (1,3,2), (2,1,3), (3,2,1). 


Those are the 3! = 6 permutations of (1,2,3); the first one is the identity. 
The determinant of A is now reduced to six separate and much simpler determinants. 
Factoring out the a;;, there is a term for every one of the six permutations: 


1 1 1 
detA = a11422a33 1 + a12@7343} 1] + 413421432 | 1 


(5) 


+ a1 1423432 1] + 12421433 |1 + 44302243} 1 
1 1 1 


Every term is a product of n = 3 entries a;;, with each row and column represented once. 
If the columns come in the order (@,...,v), that term is the product aq -+- dy, times the 
determinant of a permutation matrix P. The determinant of the whole matrix is the sum 
of these n! terms, and that sum is the explicit formula we are after: 


Big Formula detA = yy (410428 *** Any) det P. (6) 

all P’s 
For an n by n matrix, this sum is taken over all n! permutations (@,...,v) of the numbers 
(1,...,2). The permutation gives the column numbers as we go down the matrix. The is 


appear in P at the same places where the a’s appeared in A. 
It remains to find the determinant of P. Row exchanges transform it to the identity 
matrix, and each exchange reverses the sign of the determinant: 


detP =+1 or —1  foran even or odd number of row exchanges. 
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(1,3,2) is odd so 1}=-1 (3,1,2) isevenso |l1 = 
1 1 


(1,3,2) requires one exchange and (3, 1,2) requires two exchanges to recover (1,2,3). 
These are two of the six + signs. For n = 2, we only have (1,2) and (2,1): 


1 0 
detA = aj a2 det ; = 411A22 —Aj2da2,_ (or ad — be). 


0 
+ aj2a2) det ; 


No one can claim that the big formula (6) is particularly simple. Nevertheless, it is 
possible to see why it has properties 1-3. For A = J, every product of the a;; will be 
zero, except for the column sequence (1,2,...,7). This term gives det/ = 1. Property 2 
will be checked in the next section, because here we are most interested in property 3: 
The determinant should depend linearly on the first row a11,d@12,..-,@1n. 

Look at all the terms 41a42B *** Any involving a,;. The first column is @ = 1. This 
leaves some permutation (f,...,v) of the remaining columns (2,...,”). We collect all 
these terms together as a,,C 1, where the coefficient of a); is a smaller determinant— 
with row | and column | removed: 


Cofactor of a1; C= Yi (ag -+* Any) det P = det(submatrix of A). (7) 


Similarly, the entry a;2 is multiplied by some smaller determinant C)2. Grouping all the 
terms that start with the same a) ;, formula (6) becomes 


Cofactors along row 1 detA = a4;C11 + aj2Cj2 +--+ taynCin. (8) 


This shows that detA depends linearly on the entries aj1,...,@1, of the first row. 


Example 2. For a3 by 3 matrix, this way of collecting terms gives 


detA = aj1(a22a33 — a23432) + 412(a23a31 — a21@33) +.413(a21a32 — a22a31). (9) 


The cofactors C1;, C12, C13 are the 2 by 2 determinants in parentheses. 


Expansion of detA in Cofactors 


We want one more formula for the determinant. If this meant starting again from scratch, 
it would be too much, But the formula is already discovered—it is (8), and the only point 
is to identify the cofactors C, ; that multiply ay ;. 

We know that C;; depends on rows 2,...,n. Row | is already accounted for by 
a,;. Furthermore, a1; also accounts for the jth column, so its cofactor Cj ; must depend 
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entirely on the other columns. No row or column can be used twice in the same term. 
What we are really doing is splitting the determinant into the following sum: 


aj, 412 a43 ay a\2 a3 
Cofactor 
we a21 a22 a23|= a22 a23| + Ja21 a3) + |da21 a22 
splitting 
a31 432 433 a32 33 a3} a33 a3} 432 


For a determinant of order n, this splitting gives n smaller determinants (minors) of order 
n—1; you see the three 2 by 2 submatrices. The submatrix M); is formed by throwing 
away row | and column j. Its determinant is multiplied by a; ;—and by a plus or minus 
sign. These signs alternate as in detM,,, —detMj2, detM,3: 


Cofactors of row 1 C1; = (-1)'* det; ;. 


The second cofactor Cj2 iS a23a31 — d21a33, which is detM) times —1. This same tech- 
nique works on every n by n matrix. The splitting above confirms that C1; is the deter- 
minant of the lower right corner M11. 

There is a similar expansion on any other row, say row i. It could be proved by 
exchanging row i with row 1. Remember to delete row i and column j of A for Mi;: 


4B The determinant of A is a combination of any row i times its cofactors: 
detA by cofactors detA = aj)Cjj + aj2Ci2 +--+ + ainCin. (10) 
The cofactor C} ; is the determinant of M;; with the correct sign: 


delete row iandcolumn j —-C;; = (—1)'*/detMjj. (11) 


These formulas express detA as a combination of determinants of order n—1. We 
could have defined the determinant by induction on n. A 1 by 1 matrix has detA = aj, 
and then equation (10) defines the determinants of 2 by 2 matrices, 3 by 3 matrices, 
and n by n matrices. We preferred to define the determinant by its properties, which 
are much simpler to explain. The explicit formula (6) and the cofactor formula (10) 
followed directly from these properties. 

There is one more consequence of detA = detA!. We can expand in cofactors of a 
column of A, which is a row of A?. Down column j of A, 


detA = ay Cj; +42 j;Coj +++» + anjCnj. (12) 
Example 3. The 4 by 4 second difference matrix A4 has only two nonzeros in row 1: 
2 -1 0 O 
Use cofactors A4= ay en ae 


Oo =f 2 =] 
0% O°. =a <2 
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Ci; comes from erasing row | and column 1, which leaves the —1, 2, —1 pattern: 


2 -1 O 
Ci}; =detA3=det}/-1 2 -1 
0 -l1 2 
For aj2 = —1 it is column 2 that gets removed, and we need its cofactor Cj2: 
—-1 -1 0O 
142 2. = 
Ciyy=(-1) det] 0 2 -1] =+det fs = detA. 
0 -1l 2 


This left us with the 2 by 2 determinant. Altogether row 1 has produced 2C,; — C12: 
detAy = 2(detA3) — detAy = 2(4) -3 =5 
The same idea applies to As and A¢, and every A,: 
Recursion by cofactors detA, = 2(detA,_1) —detA,_2. (13) 


This gives the determinant of increasingly bigger matrices. At every step the determinant 
of A, isn-+1, from the previous determinants n and n — 1: 


—1, 2, —1 matrix detA, = 2(n) —(n—-1) =n+1. 


The answer n+ | agrees with the product of pivots at the start of this section. 


Problem Set 4.3 


1. For these matrices, find the only nonzero term in the big formula (6): 


0100 tere 
1 

re 010 a el Os at 
0101 6789 
0010 0001 


There is only one way of choosing four nonzero entries from different rows and 
different columns. By deciding even or odd, compute detA and det B. 


2. Expand those determinants in cofactors of the first row. Find the cofactors (they 
include the signs (—1)'*/) and the determinants of A and B. 


3. True or false? 


(a) The determinant of S~!AS equals the determinant of A. 


(b) If detA = O then at least one of the cofactors must be zero. 
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(c) A matrix whose entries are Os and 1s has determinant 1, 0, or —1. 
(a) Find the LU factorization, the pivots, and the determinant of the 4 by 4 matrix 
whose entries are a;; = smaller of i and j. (Write out the matrix.) 


(b) Find the determinant if a;; = smaller of n; and n;, where n| = 2, n2 = 6, n3 = 8, 
n4 = 10. Can you give a general rule for any ny < nz <n3 <1n4? 


. Let F,, be the determinant of the 1, 1, —1 tridiagonal matrix (n by n): 


By expanding in cofactors along row 1, show that F,, = F,-1 + Fy;—2. This yields the 
Fibonacci sequence 1,2,3,5,8,13,... for the determinants. 


Suppose A, is the by n tridiagonal matrix with is on the three diagonals: 


A= (1), oe 


{0 
11 
, As= ]1 1 1], 
11 

Ota 


Let D,, be the determinant of A,; we want to find it. 


(a) Expand in cofactors along the first row to show that D, = D,—1 — Dn_2. 


(b) Starting from D; = 1 and D2 = 0, find D3,D4,...,Ds. By noticing how these 
numbers cycle around (with what period?) find Dj qo. 


(a) Evaluate this determinant by cofactors of row 1: 


4444 


NO Ne 


| ieee 0 
20 1 
1 1 0 


(b) Check by subtracting column 1 from the other columns and recomputing. 


. Compute the determinants of Az, A3, Ay. Can you predict A,,? 


0; hot 4 

01 ee. 1011 

A=) 9 ail (vetoes Me og. 4 
11.0 

ile as es 


Use row operations to produce zeros, or use cofactors of row 1. 


10. 


11. 


12. 
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How many multiplications to find an n by n determinant from 

(a) the big formula (6)? 

(b) the cofactor formula (10), building from the count for n — 1? 

(c) the product of pivots formula (including the elimination steps)? 

In a5 by 5 matrix, does a + sign or — sign go with a15a24a33d42a5; down the reverse 


diagonal? In other words, is P = (5,4,3,2, 1) even or odd? The checkerboard pattern 
of + signs for cofactors does not give det P. 


If A is m by n and B isn by m, explain why 


0 A I 0 
det = detAB. Hint: Postmultiply by 
—-B I BI 


Do an example with m <n and an example with m > n. Why does your second 
example automatically have detAB = 0? 


Suppose the matrix A is fixed, except that a,; varies from —° to +e, Give examples 
in which detA is always zero or never zero. Then show from the cofactor expansion 
(8) that otherwise detA = 0 for exactly one value of aj. 


Problems 13-23 use the big formula with n! terms: |A| = Ya) 4p + -- dny. 


13. 


14. 


15. 


Compute the determinants of A, B, C from six terms. Independent rows? 


L253 L253 Bods A 
A=|3 1 2 B=|4 4 4 C=/]1 1 0 
St 2: ol a Go7 1.0: 50 


Compute the determinants of A, B, C. Are their columns independent? 
ES SPO 1) 92.3 

A= 1. 0:1 B= |4 5 6 c= 
011 7 39 


A 0 
0B 


Show that detA = 0, regardless of the five nonzeros marked by x’s: 


Me 
A= 100°. X14 (What is the rank of A?) 
0 0 x 


16. This problem shows in two ways that detA = 0 (the x’s are any numbers): 


De ae: SS 2 ik 
Ri A 5 by 5 matrix 
A=|0 00x x 3 by 3 zero matrix 
Oo 20 ae Always singular 
0 0-0 a 
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(a) How do you know that the rows are linearly dependent? 


(b) Explain why all 120 terms are zero in the big formula for detA. 


17. Find two ways to choose nonzeros from four different rows and columns: 


(B has the same zeros as A.) 


— = CO — 


2 
5 
3 
1 


orrF © 


1 0 
1 BK 4 
1 0 
1 0 


oor Oo 


Is detA equal to 1+ 1 or 1 —1 or —1 — 1? What is det B? 


18. Place the smallest number of zeros in a 4 by 4 matrix that will guarantee detA = 0. 
Place as many zeros as possible while still allowing detA ¥ 0. 


19. (a) If a), = a22 = a33 = 0, how many of the six terms in detA will be zero? 


(b) If ay) = a22 = 433 = 444 = 0, how many of the 24 products a ja2,a3¢a4m are sure 
to be zero? 


20. How many 5 by 5 permutation matrices have detP = +1? Those are even permuta- 
tions. Find one that needs four exchanges to reach the identity matrix. 


21. If detA #0, at least one of the n! terms in the big formula (6) is not zero. Deduce 
that some ordering of the rows of A leaves no zeros on the diagonal. (Don’t use P 
from elimination; that PA can have zeros on the diagonal.) 


22. Prove that 4 is the largest determinant for a 3 by 3 matrix of 1s and —1s. 


23. How many permutations of (1,2,3,4) are even and what are they? Extra credit: 
What are all the possible 4 by 4 determinants of J + Peyen? 


Problems 24-33 use cofactors C;; = (—1)'*/ detM;;. Delete row i, column j. 
24. Find cofactors and then transpose. Multiply cr and C} by A and B! 


2 1 
3: 6 


D223 
A= | B=1/4 5 6 
7 0 0 


25. Find the cofactor matrix C and compare AC! with A7!: 


2 -l O 


26. 


27. 


28. 


29. 


30. 
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The matrix B, is the —1, 2, —1 matrix A, except that bj; = 1 instead of aj; = 2. 
Using cofactors of the last row of B4, show that |B4| = 2|B3| —|Bo| = 1: 


=i. 2 


The recursion |B,,| = 2|B,—1| — |Bn—2| is the same as for the A’s. The difference is in 
the starting values 1, 1, 1 forn = 1,2,3. What are the pivots? 


B,, is still the same as A, except for bj; = 1. So use linearity in the first row, where 
[1 —1 0] equals [2 —1 0] minus [1 0 0}: 
1 -1l 0 2 —1 0 1 0 0 
B,| = —] 2 —1 _ |e 
0 0 0 
Linearity in row 1 gives |B,| = |An|— |An—1| = ___. 


The n by n determinant C,, has 1s above and below the main diagonal: 


0 1 


C= 
{6 : 


oe o| Co= 


or © 
_- Oo -_ 


0 
1 ie 
0 


oor oO 
or OF 
r- OF O&O 
or Oo Oo 


(a) What are the determinants of C), C2, C3, C4? 
(b) By cofactors find the relation between C,, and C,,_; and C,_2. Find Cj. 


Problem 28 has Is just above and below the main diagonal. Going down the matrix, 
which order of columns (if any) gives all 1s? Explain why that permutation is even 
forn = 4,8,12,... and odd for n = 2,6,10,... 


C,=O(oddn) CG,=1(n=4,8,...) C,=—1(n=2,6....). 


Explain why this Vandermonde determinant contains x° but not x* or x”: 


a ae 


be ob? 
ee ag? 
2 x? 


V4 = det 


a 
asa Oo FQ 


Xx 


The determinant is zero at x = , , and . The cofactor of x° is V3 = 
(b—a)(c—a)(c—b). Then V4 = (x —a)(x— b)(x—c)V3. 
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31. Compute the determinants $;, $2, $3 of these 1, 3, 1 tridiagonal matrices: 


a4 4°46 

5 [3| Ge eS 
1|3 2=1, 2 g= 11 3-1 
013 


Make a Fibonacci guess for S4 and verify that you are right. 


32. Cofactors of those 1, 3, 1 matrices give S, = 3S,—1 —Sp—2. Challenge: Show that 
Sn is the Fibonacci number Fyy42 by proving Fry+2 = 3Fon — Fon—2. Keep using 
Fibonacci’s rule F, = Fy_) + Fy_2. 


33. Change 3 to 2 in the upper left corner of the matrices in Problem 32. Why does 
that subtract S,_; from the determinant S,,? Show that the determinants become the 
Fibonacci numbers 2, 5, 13 (always Fon+1). 


Problems 34—36 are about block matrices and block determinants. 


34. With 2 by 2 blocks, you cannot always use block determinants! 


A B 
C D 


A B 


=|A||D| but 
0D 


# |A||D| — |CI|BI- 


(a) Why is the first statement true? Somehow B doesn’t enter. 
(b) Show by example that equality fails (as shown) when C enters. 
(c) Show by example that the answer det(AD — CB) is also wrong. 


35. With block multiplication, A = LU has A; = L;,U; in the upper left corner: 
— | 0 
7 * *k 


(a) Suppose the first three pivots of A are 2, 3, —1. What are the determinants of L), 
Ly, L3 (with diagonal 1s), U;, U2, U3, and Aj, A2, A3? 


(b) If Aj, Az, A3 have determinants 5, 6, 7, find the three pivots. 


A, * 


* * 


U, * 
O x 


y 


36. Block elimination subtracts CA~! times the first row [A B] from the second row 
[C D]. This leaves the Schur complement D—CA~'B in the corner: 


A B 
0 D—CA7!B 


I 0 
—CA-! ] 


A B 
C D 


Take determinants of these matrices to prove correct rules for square blocks: 


A B 


= |A||D —CA~'B| = |AD—CBI. 
C D 


if A~! exists ifAC=CA 
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37. A 3 by 3 determinant has three products “down to the right” and three “down to the 
left’ with minus signs. Compute the six terms in the figure to find D. Then explain 
without determinants why this matrix is or is not invertible: 


38. For Az in Problem 6, five of the 4! = 24 terms in the big formula (6) are nonzero. 
Find those five terms to show that D4 = —1. 


39. For the 4 by 4 tridiagonal matrix (entries —1, 2, —1), find the five terms in the big 
formula that give detA = 16—4-—4—4+1. 


40. Find the determinant of this cyclic P by cofactors of row 1. How many exchanges 
reorder 4, 1, 2, 3 into 1, 2, 3, 4? Is |P?| = +1 or —1? 
00 1 001 0 
fi ae lel 
100 0 I 0 
0 1 0 0 


41. A=2xeye(n)—diag(ones(n—1, 1),1)—diag(ones(n—1, 1),—1) is the —1, 2, —1 
matrix. Change A(1,1) to 1 so detA = 1. Predict the entries of A~! based on n = 3 
and test the prediction for n = 4. 


42. (MATLAB) The —1, 2, —1 matrices have determinant n+ 1. Compute (n+ 1)A7! 
for n = 3 and 4, and verify your guess for n = 5. (Inverses of tridiagonal matrices 
have the rank-1 form uv! above the diagonal.) 


43. All Pascal matrices have determinant 1. If I subtract 1 from the n, n entry, why does 
the determinant become zero? (Use rule 3 or a cofactor.) 


Eo: ote | a | 
12 3 4 123 4 

det oe ey = | (known) det (8 e406 = 0 (explain). 
1 4 10 20 1 4 10 19 


4.4 Applications of Determinants 


This section follows through on four major applications: inverse of A, solving Ax = b, 
volumes of boxes, and pivots. They are among the key computations in linear algebra 
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(done by elimination). Determinants give formulas for the answers. 


1. Computation of A~!. The 2 by 2 case shows how cofactors go into A~!: 


= 
a b _ 1 d- =D). --- 3 
c d|  ad—be|—c a|_ deta 
We are dividing by the determinant, and A is invertible exactly when detA is nonzero. 
The number C ; = d is the cofactor of a. The number C)2 = —c is the cofactor of b (note 
the minus sign). That number C)2 goes in row 2, column 1! 


The row a, b times the column C1, Ci2 produces ad — bc. This is the cofactor expan- 
sion of detA. That is the clue we need: A! divides the cofactors by det A. 


Cir Car 
Cio Cy 


3 iT . 
Cistrampeed «2 aaa me Ama OD 
Our goal is to verify this formula for A~!. We have to see why AC! = (det A)I: 
fi (ee ee Po ee es dettA -:- O 
hp Se ae (2) 
Bee tie ll Oxo. Bee SS 0 .-- detA 
With cofactors Cj1,...,C1, 1n the first column and not the first row, they multiply a11,...,@1n 


and give the diagonal entry detA. Every row of A multiplies its cofactors (the cofactor 
expansion) to give the same answer detA on the diagonal. 

The critical question is: Why do we get zeros off the diagonal? If we combine the 
entries a); from row | with the cofactors C2; for row 2, why is the result zero? 


row | of A, row 2 of C a11C21 +442Cr2 + +++ tajnCoy = 0. (3) 


The answer is: We are computing the determinant of a new matrix B, with a new row 2. 
The first row of A is copied into the second row of B. Then B has two equal rows, and 
det B = 0. Equation (3) is the expansion of det B along its row 2, where B has exactly 
the same cofactors as A (because the second row is thrown away to find those cofactors). 
The remarkable matrix multiplication (2) is correct. 

That multiplication AC! = (detA)/J immediately gives A~'. Remember that the cofac- 
tor from deleting row i and column j of A goes into row j and column i of C’. Dividing 
by the number detA (if it is not zero!) gives A~! = C™/detA. 


Example 1. The inverse of a sum matrix is a difference matrix: 


The minus signs enter because cofactors always include (—1)'*/. 
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2. The Solution of Ax = b. The multiplication x = A~'b is just C'b divided bydetA. 
There is a famous way in which to write the answer (x1,...,Xn): 


4C Cramer’s rule: The jth component of x = A~'b is the ratio 
ayy a2 by ay 
det B; " 
5 : 
J detA 


where Bj;=| : : : : | has bincolumn j. (4) 


Ant Gnd Dy. Ginn 


Proof. Expand det B; in cofactors of its jth column (which is b). Since the cofactors 
ignore that column, det B; is exactly the jth component in the product C'b: 


det Bj = bi Cy; + b2C2; + +++ + dnCy;- 


Dividing this by detA gives x;. Each component of x is a ratio of two determinants. That 
fact might have been recognized from Gaussian elimination, but it never was. 


Example 2. The solution of 


Xp = Sx HON 
2x, + 4%. = 6 


has O and 6 in the first column for x; and in the second column for x2: 


0 3 1 O 
6 4, _18 2 6 6 
xj, = — =. x2 = > — = — = -3. 
13) 2 13) 2 
2 4 2 4 


The denominators are always detA. For 1000 equations Cramer’s Rule would need 1001 
determinants. To my dismay I found in a book called Mathematics for the Millions that 
Cramer’s Rule was actually recommended (and elimination was thrown aside): 


To deal with a set involving the four variables u, v, w, z, we first have to 
eliminate one of them in each of three pairs to derive three equations in three 
variables and then proceed as for the three-fold left-hand set to derive values 
for two of them. The reader who does so as an exercise will begin to realize 
how formidably laborious the method of elimination becomes, when we have 
to deal with more than three variables. This consideration invites us to explore 
the possibility of a speedier method... 


3. The Volume of a Box. The connection between the determinant and the volume is 
clearest when all angles are right angles—the edges are perpendicular, and the box is 
rectangular. Then the volume is the product of the edge lengths: volume = €,02---y. 
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We want to obtain the same ¢;02---¢, from detA, when the edges of that box are the 
rows of A. With right angles, these rows are orthogonal and AA! is diagonal: 


row | 


Zon 


Right-angled box Pre 
Orthogonal rows 


| 
me SOR 


= 
i) 


row n e 


The ¢; are the lengths of the rows (the edges). and the zeros off the diagonal come 
because the rows are orthogonal. Using the product and transposing rules, 


Rightangle case 05 ---2 = det(AA‘) = (detA)(detA') = (detA)?. 


The square root of this equation says that the determinant equals the volume. The sign 
of detA will indicate whether the edges form a “right-handed” set of coordinates, as in 
the usual x-y-z system, or a left-handed system like y-x-z. 

If the angles are not 90°, the volume is not the product of the lengths. In the plane 
(Figure 4.2), the “volume” of a parallelogram equals the base ¢ times the height , The 
vector b — p of length h is the second row b = (a21,d22), minus its projection p onto the 
first row. The key point is this: By rule 5, detA is unchanged when a multiple of row 1 
is subtracted from row 2. We can change the parallelogram to a rectangle, where it is 
already proved that volume = determinant. 

In n dimensions, it takes longer to make each box rectangular, but the idea is the 
same. The volume and determinant are unchanged if we subtract from each row its pro- 
jection onto the space spanned by the preceding rows—leaving a perpendicular “height 
vector” like pb. This Gram-Schmidt process produces orthogonal rows, with volume = 
determinant. So the same equality must have held for the original rows. 


‘ teasaeg | dan? 
\ height i ee : A 


Figure 4.2: Volume (area) of the parallelogram = @ times h = | det A]. 


This completes the link between volumes and determinants, but it is worth coming 
back one more time to the simplest case. We know that 


det E ‘ =a det } = 
0 1 c l 
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These determinants give the volumes—or areas, since we are in two dimensions—drawn 
in Figure 4.3. The parallelogram has unit base and unit height; its area is also 1. 


row 2 = (c, 1) 


row 1 = (1,0) 


Figure 4.3: The areas of a unit square and a unit parallelogram are both 1. 


4. A Formula for the Pivots. We can finally discover when elimination is possible 
without row exchanges. The key observation is that the first k pivots are completely 
determined by the submatrix A, in the upper left corner of A. The remaining rows and 
columns of A have no effect on this corner of the problem: 


Elimination on A a be a b e 
includes A=|c d f|—|0 (ad—bc)/a (af —ec)/a 
elimination on A> gh i g h i 


Certainly the first pivot depended only on the first row and column, The second pivot 
(ad — bc) /a depends only on the 2 by 2 corner submatrix A>. The rest of A does not enter 
until the third pivot. Actually it is not just the pivots, but the entire upper-left corners of 
L, D, and U, that are determined by the upper-left corner of A: 


1 a 1 b/a x 
A=LDU=|c/a 1 (ad —bc)/a 1 x 
* * I * 1 


What we see in the first two rows and columns is exactly the factorization of the corner 
submatrix Az. This is a general rule if there are no row exchanges: 


4D If A is factored into LDU, the upper left corners satisfy Ay, = L,D,U,. For 
every k, the submatrix A; is going through a Gaussian elimination of its own. 


The proof is to see that this corner can be settled first, before even looking at other 
eliminations. Or use the laws for block multiplication: 

Ty, 0 
BC 


Dy, 0 
0 E 


U, F 
0 G 


Ly,D,U;E L,D,F 
BD,U, BD,F +CEG| 


LDU = 


Comparing the last matrix with A, the corner L;D,U; coincides with A;. Then: 


det A; = detL; det D; det U; = det Dy = di d2--- dk. 
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The product of the first k pivots is the determinant of Ax. This is the same rule that 
we know already for the whole matrix. Since the determinant of A,_; will be given by 
d\dz---dg_1, we can isolate each pivot d; as a ratio of determinants: 


4 det A, d\dz---dx 
Formula for pivots rey ae = Fras =p. (5) 
In our example above, the second pivot was exactly this ratio (ad — bc)/a. It is the 
determinant of Az divided by the determinant of A;. (By convention det Ag = 1, so that 
the first pivot is a/1 = a.) Multiplying together all the individual pivots, we recover 
detA, detA> detA, detA, 
d\dz-+-dy = fe = 
detAo detA, detA,_ det Ag 
From equation (5) we can finally read off the answer to our original question: The 
pivot entries are all nonzero whenever the numbers det A, are all nonzero: 


= detA. 


4E Elimination can be completed without row exchanges (so P = J and A = 
LU), if and only if the leading submatrices A;,A2,...,A, are all nonsingular. 


That does it for determinants, except for an optional remark on property 2—the sign 
reversal on row exchanges. The determinant of a permutation matrix P was the only 
questionable point in the big formula. Independent of the particular row exchanges link- 
ing P to J, is the number of exchanges always even or always odd? If so, its determinant 
is well defined by rule 2 as either +1 or —1. 

Starting from (3,2,1), a single exchange of 3 and | would achieve the natural order 
(1,2,3). So would an exchange of 3 and 2, then 3 and 1, and then 2 and 1. In both 
sequences, the number of exchanges is odd. The assertion is that an even number of 
exchanges can never produce the natural order beginning with (3,2, 1). 

Here is a proof. Look at each pair of numbers in the permutation, and let N count 
the pairs in which the larger number comes first. Certainly VN = 0 for the natural order 
(1,2,3). The order (3,2, 1) has N = 3 since all pairs (3,2), (3,1), and (2,1) are wrong. 
We will show that every exchange alters N by an odd number. Then to arrive at N = 0 
(the natural order) takes a number of exchanges having the same evenness or oddness as 
N. 

When neighbors are exchanged, N changes by +1 or —1. Any exchange can be 
achieved by an odd number of exchanges of neighbors. This will complete the proof; 
an odd number of odd numbers is odd. To exchange the first and fourth entries below, 
which happen to be 2 and 3, we use five exchanges (an odd number) of neighbors: 


(2,1,4,3) — (1,2,4,3) — (1,4,2,3) — (1,4,3,2) — (1,3,4,2) — (3,1,4,2). 


We need ¢—k exchanges of neighbors to move the entry in place k to place @. Then 
¢—k-—1 exchanges move the one originally in place ¢ (and now found in place ¢— 1) 
back down to place k. Since (€—k) + (€—k-—1) is odd, the proof is complete. The 
determinant not only has all the properties found earlier, it even exists. 
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Problem Set 4.4 


1. Find the determinant and all nine cofactors C;; of this triangular matrix: 


Form C! and verify that AC’ = (detA)/. What is A~!? 


2. Use the cofactor matrix C to invert these symmetric matrices: 


2 -1 0O 111 
A= ]-1 2 -l and B= pl) 22 
0 -1 2 LS 
3. Find x, y, and z by Cramer’s Rule in equation (4): 
4y — =.“ 
ax + by = 1 ie ‘ 
ds Gi 0 and Seo ABS BS 0 
cx = 
2X + 3z = 0. 


4. (a) Find the determinant when a vector x replaces column j of the identity (consider 
Xx; = 0 as a separate case): 


if M= xj then detM = 


XA 1 
(b) If Ax = b, show that AM is the matrix B; in equation (4), with b in column j. 
(c) Derive Cramer’s rule by taking determinants in AM = B;. 
5. (a) Draw the triangle with vertices A = (2,2), B = (—1,3), and C = (0,0). By 
regarding it as half of a parallelogram, explain why its area equals 


1 2 DZ 
area(ABC) = 5 det | ; ; 


(b) Move the third vertex to C = (1, —4) and justify the formula 


x; yy 1 2 2 1 
area(ABC) = 5 det x. yo Il = 5 det saly 432 - Of 
x3 y3 | 1 -4 1 
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Hint: Subtracting the last row from each of the others leaves 


D2. D> if 1 6 0 
1 6 
det}—1 3 1] =det|—2 7 O|] =det 5 al 
1 -4 1 1 -4 1 


Sketch A’ = (1,6), B’ = (—2,7), C’ = (0,0) and their relation to A, B, C. 
6. Explain in terms of volumes why det3A = 3”detA for an n by n matrix A. 


7. Predict in advance, and confirm by elimination, the pivot entries of 


8. Find all the odd permutations of the numbers {1,2,3,4}. They come from an odd 
number of exchanges and lead to det P = —1. 


9. Suppose the permutation P takes (1,2,3,4,5) to (5,4,1,2,3). 


(a) What does P? do to (1,2,3,4,5)? 
(b) What does P~! do to (1,2,3,4,5)? 


10. If P is an odd permutation, explain why P? is even but P~! is odd. 


11. Prove that if you keep multiplying A by the same permutation matrix P, the first row 
eventually comes back to its original place. 


12. If A is a5 by 5 matrix with all |a;;| < 1, then detA < . Volumes or the big 
formula or pivots should give some upper bound on the determinant. 


Problems 13-17 are about Cramer’s Rule for x = A~'b. 


13. Solve these linear equations by Cramer’s Rule x; = det B;/detA: 


2x, + 5 1 a =x 
X X = 
(a) ’ . (b) xy + 2x. + x3 = 70 


4 
ss a x. + 2x3 = O. 


| 
ES 


14. Use Cramer’s Rule to solve for y (only). Call the 3 by 3 determinant D: 


ax = by Sb. ez =. 1 
(b) dx + ey — fz = 0 
ex + hy + iz = 0. 


— 


ax +> by = 


o cx + dy = 0 
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15. Cramer’s Rule breaks down when detA = 0. Example (a) has no solution, whereas 
(b) has infinitely many. What are the ratios x; = det B;/detA? 


2x, +3x2=1 : 2x; +3x2=1 ; 
(parallel lines) (b) (same line) 
4x, + 6x2 = 1. 


(a) 4x; + 6x2 = 2. 


16. Quick proof of Cramer’s rule. The determinant is a linear function of column 1. It is 
zero if two columns are equal. When b = Aa = x, a1 +x2a2 +.x3@3 goes into column 
1 to produce B,, the determinant is 


U a2 as| = vay +x2a2+%x%3a3 a2 as| =x lay a2 as| = x; detA. 
(a) What formula for x; comes from left side = right side? 
(b) What steps lead to the middle equation? 


17. If the right side b is the last column of A, solve the 3 by 3 system Ax = b. Explain 
how each determinant in Cramer’s Rule leads to your solution x. 


Problems 18-26 are about A~' = C'/detA. Remember to transpose C. 


18. Find A~! from the cofactor formula C'/detA. Use symmetry in part (b): 


Lt 20 2 -l O 
(a) A=|0 3 0 (b) A=|-1 2 -1 
04 1 Oh. 2 


19. If all the cofactors are zero, how do you know that A has no inverse? If none of the 
cofactors are zero, is A sure to be invertible? 


20. Find the cofactors of A and multiply AC’ to find det A: 


4 6 —3 0 
ON: C=]. - -|, and ACT = 
5 ‘ 


If you change that corner entry from 4 to 100, why is detA unchanged? 
21. Suppose detA = | and you know all the cofactors. How can you find A? 
22. From the formula AC! = (detA)/ show that detC = (detA)""!. 


23. (For professors only) If you know all 16 cofactors of a 4 by 4 invertible matrix A, 
how would you find A? 


24. If all entries of A are integers, and detA = 1 or —1, prove that all entries of A~! are 
integers. Give a 2 by 2 example. 
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25. L is lower triangular and S is symmetric. Assume they are invertible: 


a 0 0 abd 
LE =-|)b.<c 10 S = lie eZ 
des def 


(a) Which three cofactors of L are zero? Then L~! is lower triangular. 


(b) Which three pairs of cofactors of § are equal? Then S~! is symmetric. 


26. For n = 5 the matrix C contains cofactors and each 4 by 4 cofactor contains 
terms and each term needs multiplications. Compare with 5° = 125 for 
the Gauss-Jordan computation of A7!. 


Problems 27-36 are about area and volume by determinants. 
27. (a) Find the area of the parallelogram with edges v = (3,2) and w = (1,4). 


(b) Find the area of the triangle with sides v, w, and v+ w. Draw it. 


(c) Find the area of the triangle with sides v, w, and w — v. Draw it. 


28. A box has edges from (0,0,0) to (3,1,1), (1,3,1), and (1, 1,3). Find its volume and 
also find the area of each parallelogram face. 


29. (a) The corners of a triangle are (2,1), (3,4), and (0,5). What is the area? 


(b) A new comer at (—1,0) makes it lopsided (four sides). Find the area. 


30. The parallelogram with sides (2,1) and (2,3) has the same area as the parallelogram 
with sides (2,2) and (1,3). Find those areas from 2 by 2 determinants and say why 
they must be equal. (I can’t see why from a picture. Please write to me if you do.) 


31. The Hadamard matrix H has orthogonal rows. The box is a hypercube! 


Ii te. lb* xd 
1 1 -1 -1 eee 
What is detH = poe oe Yel volume of a hypercube in R"? 
1-1 1 -1 


32. If the columns of a 4 by 4 matrix have lengths L), Lo, L3, L4, what is the largest 
possible value for the determinant (based on volume)? If all entries are 1 or —1, 
what are those lengths and the maximum determinant? 


33. Show by a picture how a rectangle with area x;y2 minus a rectangle with area x2) 
produces the area x;y2 — x2y; of a parallelogram. 


34. When the edge vectors a, b, c are perpendicular, the volume of the box is ||a|| times 
||b|| times ||c|]. The matrix ATA is . Find detA‘A and det A. 


35. 


36. 
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An n-dimensional cube has how many corners? How many edges? How many (n — 
1)-dimensional faces? The n-cube whose edges are the rows of 27 has volume __. 
A hypercube computer has parallel processors at the corners with connections along 
the edges. 


The triangle with corners (0,0), (1,0), (0,1) has area 5. The pyramid with four 
corners (0,0,0), (1,0,0), (0,1,0), (0,0,1) has volume . The pyramid in R* 


with five corners at (0,0,0,0) and the rows of J has what volume? 


Problems 37—40 are about areas dA and volumes dV in calculus. 


37. 


38. 


39. 


40. 


41. 


42. 


43. 


Polar coordinates satisfy x = rcos @ and y = rsin@. Polar area J dr d@ includes J: 


cos@ —rsin@ 


Ox/dr dx/d0 
dy/dr dy/d@ 


sin@ rcos@ 


The two columns are orthogonal. Their lengths are . Thus J = 


Spherical coordinates p, @, 0 give x = psing@cos8, y= psingsin#@, z= pcosd@. 
Find the Jacobian matrix of 9 partial derivatives: 0x/dp, 0x/00, 0x/00 are in row 
1. Simplify its determinant to J = p?sin@. Then dV = p’sin@ dp dé dé. 


The matrix that connects r, 8 to x, y is in Problem 37. Invert that matrix: 


Or/dx or/dy 
00/dx d0/dy 


cos@ ? 
? 4 


ip = =? 


It is surprising that 0r/0x = 0x/dr. The product JJ~! = I gives the chain rule 


Ox | OxOr | 0x00 | 
Ox ordox Od0d0x 


The triangle with corners (0,0), (6,0), and (1,4) has area . When you rotate it 
by 0 = 60° the area is . The rotation matrix has 
— sj ae: 
determinant = moe ree =—|2 |=? 
sin@ cos@ ae 
Let P = (1,0,—1), Q = (1,1,1), and R = (2,2,1). Choose S so that PORS is a 


parallelogram, and compute its area. Choose 7, U, V so that OPORSTUYV is a tilted 
box, and compute its volume. 


Suppose (x,y,z), (1,1,0), and (1,2, 1) lie on a plane through the origin. What deter- 
minant is zero? What equation does this give for the plane? 


Suppose (x,y,z) is a linear combination of (2,3,1) and (1,2,3). What determinant 
is zero? What equation does this give for the plane of all combinations? 
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44, If Ax = (1,0,...,0) show how Cramer’s Rule gives x = first column of A~'. 


45. (VISA to AVIS) This takes an odd number of exchanges (IVSA, AVSI, AVIS). Count 
the pairs of letters in VISA and AVIS that are reversed from alphabetical order. The 
difference should be odd. 


Review Exercises 


4.1 Find the determinants of 


and 


— — — — 
hE Se 
= WO — = 
=a BS NO 
— 
N 
— 
So 


4.2 If B=M~—'!AM, why is detB = det A? Show also that detA~!B = 1. 


4.3 Starting with A, multiply its first row by 3 to produce B, and subtract the first row 
of B from the second to produce C. How is detC related to detA? 


4.4 Solve 3u+ 2v = 7, 4u+3v = 11 by Cramer’s rule. 


4.5 If the entries of A and A~! are all integers, how do you know that both determinants 
are 1 or —1? Hint: What is detA times detA~!? 


4.6 Find all the cofactors, and the inverse or the nullspace, of 
ae) cos@ —sin®@ ab 
, F , and ; 
6 9 sin@ cos@ a b 


4.7 What is the volume of the parallelepiped with four of its vertices at (0,0,0), (—1,2,2), 
(2,—1,2), and (2,2,—1)? Where are the other four vertices? 


4.8 How many terms are in the expansion of a 5 by 5 determinant, and how many are 
sure to be zero if a2; = 0? 


4.9 If P, is an even permutation matrix and P) is odd, deduce from P; + P, = P, (Pi aa 
P})P» that det(P; + P)) = 0. 


4.10 If detA > 0, show that A can be connected to J by a continuous chain of matrices 
A(t) all with positive determinants. (The straight path A(t) = A++t(J —A) does go 
from A(0) =A to A(1) = J, but in between A(t) might be singular. The problem is 
not so easy, and solutions are welcomed by the author.) 


4.11 


4.12 


4.13 


4.14 


4.15 


4.16 


4.17 
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Explain why the point (x,y) is on the line through (2,8) and (4,7) if 


x y 
det |2 8 0, or x+2y—18=0. 
4 7 


aa 
I| 


In analogy with the previous exercise, what is the equation for (x,y,z) to be on the 
plane through (2,0,0), (0,2,0), and (0,0,4)? It involves a 4 by 4 determinant. 


If the points (x,y,z), (2,1,0), and (1,1,1) lie on a plane through the origin, what 
determinant is zero? Are the vectors (1,0,—1), (2,1,0), (1,1, 1) independent? 


If every row of A has either a single +1, or a single —1, or one of each (and is 
otherwise zero), show that detA = | or —1 or 0. 


If C = [45] and D = [j, 2], then CD = —DC yields 4 equations Ax = 0: 


2a ec b 0 Uu 
b a+d_ 0 b v 
Cc 0 a+d c||w 
0 Cc be “2d iz 


CD+DC=0 is 


ooo Oo 


(a) Show that detA = 0 if a+ d=0. Solve for u, v, w, z, the entries of D. 
(b) Show that detA = 0 if ad = bc (so C is singular). 


In all other cases, CD = —DC is only possible with D = zero matrix. 


The circular shift permutes (1,2,...,7) into (2,3,...,1). What is the corresponding 
permutation matrix P, and (depending on n) what is its determinant? 


Find the determinant of A = eye(5) + ones(5) and if possible eye(n) + ones(n). 


Eigenvalues and Eigenvectors 


5.1. Introduction 


This chapter begins the “second half” of linear algebra. The first half was about Ax = 
b. The new problem Ax = Ax will still be solved by simplifying a matrix—making it 
diagonal if possible. The basic step is no longer to subtract a multiple of one row from 
another: Elimination changes the eigenvalues, which we don’t want. 

Determinants give a transition from Ax = b to Ax = Ax. In both cases the determinant 
leads to a “formal solution”: to Cramer’s rule for x = A~'b, and to the polynomial 
det(A — AJ), whose roots will be the eigenvalues. (All matrices are now square; the 
eigenvalues of a rectangular matrix make no more sense than its determinant.) The 
determinant can actually be used if n = 2 or 3. For large n, computing A is more difficult 
than solving Ax = b. 

The first step is to understand how eigenvalues can be useful, One of their applications 
is to ordinary differential equations. We shall not assume that the reader is an expert on 
differential equations! If you can differentiate x”, sinx, and e*, you know enough. As a 
specific example, consider the coupled pair of equations 


ico v=8 at ¢=0, 

dt (1) 
dw 

—=2v—3w, w=S5 at t=0. 

dt 


This is an initial-value problem. The unknown is specified at time t = 0 by the given 
initial values 8 and 5. The problem is to find v(t) and w(t) for later times t > 0. 

It is easy to write the system in matrix form. Let the unknown vector be u(r), with 
initial value u(0). The coefficient matrix is A: 


t 8 4 —5 
Vector unknown u(t) = uy) , u(0)= , A= 
w(t) 5 2 -3 
The two coupled equations become the vector equation we want: 
d 
Matrixform = —~=Au with u=u(0) at r=0. (2) 


dt 
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This is the basic statement of the problem. Note that it is a first-order equation—no 
higher derivatives appear—and it is Jinear in the unknowns, It also has constant coeffi- 
cients; the matrix A is independent of time. 

How do we find u(t)? If there were only one unknown instead of two, that question 
would be easy to answer. We would have a scalar instead of a vector equation: 


d 
Single equation | =au with u=u(0) at t=0. (3) 
The solution to this equation is the one thing you need to know: 
Pure exponential —u(t) = e“u(0). (4) 


At the initial time t = 0, u equals u(0) because e? = 1. The derivative of e“ has the 
required factor a, so that du/dt = au. Thus the initial condition and the equation are 
both satisfied. 

Notice the behavior of u for large times. The equation is unstable if a > 0, neutrally 
stable if a = 0, or stable if a < 0; the factor e“ approaches infinity, remains bounded, 
or goes to zero. If a were a complex number, a = a@ + if, then the same tests would be 
applied to the real part ~. The complex part produces oscillations e!Bt — cog Bt+isin Bt. 
Decay or growth is governed by the factor e™’. 

So much for a single equation. We shall take a direct approach to systems, and look 
for solutions with the same exponential dependence on t just found in the scalar case: 


v(t) = eMy 
es (5) 
w(t) =e”'z 
or in vector notation 
u(t) = e*'x. (6) 


This is the whole key to differential equations du/dt = Au: Look for pure exponential 
solutions. Substituting v = e*y and w = e*"z into the equation, we find 


del y = 4ey —5e"z 
Nez = Ley —3eMz, 


The factor e”’ is common to every term, and can be removed. This cancellation is the 
reason for assuming the same exponent A for both unknowns; it leaves 


4y—5z=Ay 


Eigenvalue problem 
ss 2y —3z=Az. 


(7) 


That is the eigenvalue equation. In matrix form it is Ax = Ax. You can see it again if we 
use u = e*'x—a number e*’ that grows or decays times a fixed vector x. Substituting 


into du/dt = Au gives Ae*x = Ae*x. The cancellation of e*' produces 


Eigenvalue equation Ax = Ax. (8) 
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Now we have the fundamental equation of this chapter. It involves two unknowns 
A and x. It is an algebra problem, and differential equations can be forgotten! The 
number A (lambda) is an eigenvalue of the matrix A, and the vector x is the associated 
eigenvector. Our goal is to find the eigenvalues and eigenvectors, A’s and x’s, and to use 
them. 


The Solution of Ax = Ax 


Notice that Ax = Ax is a nonlinear equation; A multiplies x. If we could discover A, then 
the equation for x would be linear. In fact we could write A/x in place of Ax, and bring 
this term over to the left side: 

(A —ADx =0. (9) 


The identity matrix keeps matrices and vectors straight; the equation (A — A )x = 0 is 
shorter, but mixed up. This is the key to the problem: 


The vector x is in the nullspace of A — XI. 
The number dX is chosen so that A — AI has a nullspace. 


Of course every matrix has a nullspace. It was ridiculous to suggest otherwise, but you 
see the point. We want a nonzero eigenvector x, The vector x = 0 always satisfies 
Ax = Ax, but it is useless in solving differential equations. The goal is to build u(t) out 
of exponentials e*"x, and we are interested only in those particular values A for which 
there is a nonzero eigenvector x. To be of any use, the nullspace of A — AJ must contain 
vectors other than zero. In short, A — AJ must be singular. 

For this, the determinant gives a conclusive test. 


5A The number A is an eigenvalue of A if and only if A — A/ is singular: 
det(A — AI) = 0. (10) 
This is the characteristic equation. Each A is associated with eigenvectors x: 
(A—ADx=0 or Ax = Ax. (11) 
In our example, we shift A by AJ to make it singular: 


4 — és 
SubtractAJ A-—Al= ‘ 2 
2 -3-A 
Note that A is subtracted only from the main diagonal (because it multiplies /). 
Determinant |A —Al| = (4—A)(—3-—-A) +10 or Maha? 


This is the characteristic polynomial. Its roots, where the determinant is zero, are the 
eigenvalues. They come from the general formula for the roots of a quadratic, or from 
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factoring into A* A —2 =(A+1)(A —2). That is zero if A = —1 or A = 2, as the 
general formula confirms: 


—b+ Vb? —4ac _ 1+V/9 
2a re) 


There are two eigenvalues, because a quadratic has two roots. Every 2 by 2 matrix 
A—ATIhas A? (and no higher power of 1) in its determinant. 

The values A = —1 and A = 2 lead to a solution of Ax = Ax or (A—AI)x =0. A matrix 
with zero determinant is singular, so there must be nonzero vectors x in its nullspace. In 
fact the nullspace contains a whole Jine of eigenvectors; it is a subspace! 


pa Uae ; | i 2 RE 


The solution (the first eigenvector) is any nonzero multiple of x: 


Eigenvalues A= = —] and 2. 


1 
Eigenvector for A, xj= | ; 


The computation for Az is done separately: 


nar acanee(t “P= 


The second eigenvector is any nonzero multiple of x2: 
, 5 
Eigenvector for A x2 = el 


You might notice that the columns of A — A,/ give x2, and the columns of A — Aj/ are 
multiples of x;. This is special (and useful) for 2 by 2 matrices. 

In the 3 by 3 case, I often set a component of x equal to | and solve (A — AJ)x = 0 for 
the other components. Of course if x is an eigenvector then so is 7x and so is —x. All 
vectors in the nullspace of A — AJ (which we call the eigenspace) will satisfy Ax = Ax. 
In our example the eigenspaces are the lines through x, = (1,1) and x2 = (5,2). 

Before going back to the application (the differential equation), we emphasize the 
steps in solving Ax = Ax: 

1. Compute the determinant of A— AI. With A subtracted along the diagonal, this 


determinant is a polynomial of degree n. It starts with (—A)”. 


2. Find the roots of this polynomial. The n roots are the eigenvalues of A. 


3. For each eigenvalue solve the equation (A — AI)x = 0. Since the determinant is 
zero, there are solutions other than x = 0. Those are the eigenvectors. 
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In the differential equation, this produces the special solutions u = ex. They are the 
pure exponential solutions to du/dt = Au. Notice e~ and e~": 


u(t) = eM" x, =e! | aud u(t) = ety, a5 ; 


These two special solutions give the complete solution. They can be multiplied by any 
numbers c; and c2, and they can be added together. When wu, and wz satisfy the linear 
equation du/dt = Au, so does their sum uw + U2: 


Complete solution u(t) = cpex, + cpe2'xy (12) 


This is superposition, and it applies to differential equations (homogeneous and linear) 
just as it applied to matrix equations Ax = 0. The nullspace is always a subspace, and 
combinations of solutions are still solutions. 

Now we have two free parameters c; and c2, and it is reasonable to hope that they can 
be chosen to satisfy the initial condition u = u(0) at t = 0: 


1 
Initial condition = cjxj +oox2 =u(0) or >} yer} 8) gay 
1 2] |e 5 


The constants are cy = 3 and cz = 1, and the solution to the original equation is 


; : (14) 


ot 
+e 
1 


t)=3e" 
u(t) e A 


Writing the two components separately, we have v(0) = 8 and w(0) = 5S: 
Solution —v(t) = 3e- + Se”, w(t) = 3e* + 2e*, 


The key was in the eigenvalues A and eigenvectors x. Eigenvalues are important in 
themselves, and not just part of a trick for finding u. Probably the homeliest example 
is that of soldiers going over a bridge.!' Traditionally, they stop marching and just walk 
across. If they happen to march at a frequency equal to one of the eigenvalues of the 
bridge, it would begin to oscillate. (Just as a child’s swing does; you soon notice the 
natural frequency of a swing, and by matching it you make the swing go higher.) An 
engineer tries to keep the natural frequencies of his bridge or rocket away from those of 
the wind or the sloshing of fuel. And at the other extreme, a stockbroker spends his life 
trying to get in line with the natural frequencies of the market. The eigenvalues are the 
most important feature of practically any dynamical system. 


Summary and Examples 


To summarize, this introduction has shown how A and x appear naturally and auto- 
matically when solving du/dt = Au. Such an equation has pure exponential solutions 


‘One which I never really believed—but a bridge did crash this way in 1831. 
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u = e*'x; the eigenvalue gives the rate of growth or decay, and the eigenvector x develops 
at this rate. The other solutions will be mixtures of these pure solutions, and the mixture 
is adjusted to fit the initial conditions. 

The key equation was Ax = Ax. Most vectors x will not satisfy such an equation. 
They change direction when multiplied by A, so that Ax is not a multiple of x. This 
means that only certain special numbers are eigenvalues, and only certain special 
vectors x are eigenvectors. We can watch the behavior of each eigenvector, and then 
combine these “normal modes” to find the solution. To say the same thing in another 
way, the underlying matrix can be diagonalized. 

The diagonalization in Section 5.2 will be applied to difference equations, Fibonacci 
numbers, and Markov processes, and also to differential equations. In every example, 
we start by computing the eigenvalues and eigenvectors; there is no shortcut to avoid 
that. Symmetric matrices are especially easy. “Defective matrices” lack a full set of 
eigenvectors, so they are not diagonalizable. Certainly they have to be discussed, but we 
will not allow them to take over the book. 

We start with examples of particularly good matrices. 


Example 1. Everything is clear when A is a diagonal matrix: 


red 


1 0 
2 has A; =3 with x= ‘ Ayg=2 with m= F 
2 0 1 


On each eigenvector A acts like a multiple of the identity: Ax; = 3x; and Ax2 = 2x2. 
Other vectors like x = (1,5) are mixtures x; + 5x2 of the two eigenvectors, and when A 
multiplies x; and x it produces the eigenvalues A; = 3 and Az = 2: 


3 
A times x;+5x2 is an 410%= |] 


This is Ax for a typical vector x—not an eigenvector. But the action of A is determined 
by its eigenvectors and eigenvalues. 


Example 2. The eigenvalues of a projection matrix are | or 0! 


P= 


NENTS 
NIE 


1 1 
| has A,;=1 with a=) Ar =0 with n=| i, 


We have A = 1| when x projects to itself, and A = 0 when x projects to the zero vector. 
The column space of P is filled with eigenvectors, and so is the nullspace. If those spaces 
have dimension r and n—r, then A = 1 is repeated r times and A = 0 is repeated n —r 
times (always n ’s): 


F i l 
our ewenya ues has A =1,1,0,0. 
allowing repeats 


ooo Fr 
ooo Oo 
ooo oO 
re OO CO 
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There is nothing exceptional about 1 = 0. Like every other number, zero might be 
an eigenvalue and it might not. If it is, then its eigenvectors satisfy Ax = Ox. Thus x is 
in the nullspace of A. A zero eigenvalue signals that A is singular (not invertible); its 
determinant is zero. Invertible matrices have all A 4 0. 


Example 3. The eigenvalues are on the main diagonal when A is triangular: 


1-A 
det(A—AI) =| 0 
0 0 


K 


5 
-A 6 |=(1-A)(Z-A)G—A). 
I 
5A 
The determinant is just the product of the diagonal entries. It is zero if A = 1, A = ;, or 
A= x the eigenvalues were already sitting along the main diagonal. 


This example, in which the eigenvalues can be found by inspection, points to one 
main theme of the chapter: To transform A into a diagonal or triangular matrix without 
changing its eigenvalues. We emphasize once more that the Gaussian factorization A = 
LU is not suited to this purpose. The eigenvalues of U may be visible on the diagonal, 
but they are not the eigenvalues of A. 

For most matrices, there is no doubt that the eigenvalue problem is computationally 
more difficult than Ax = b. With linear systems, a finite number of elimination steps 
produced the exact answer in a finite time. (Or equivalently, Cramer’s rule gave an exact 
formula for the solution.) No such formula can give the eigenvalues, or Galois would 
turn in his grave. For a 5 by 5 matrix, det(A — A/) involves A>. Galois and Abel proved 
that there can be no algebraic formula for the roots of a fifth-degree polynomial. 

All they will allow is a few simple checks on the eigenvalues, after they have been 
computed, and we mention two good ones: sum and product. 


5B The sum of the n eigenvalues equals the sum of the n diagonal entries: 
Traceof A=A,4+---+A,=ay+--: tan. (15) 


Furthermore, the product of the n eigenvalues equals the determinant of A. 


The projection matrix P had diagonal entries 5, 4 and eigenvalues 1, 0. Then s+ 5 


agrees with 1+ 0 as it should. So does the determinant, which is 0-1 =0. A singular 
matrix, with zero determinant, has one or more of its eigenvalues equal to zero. 

There should be no confusion between the diagonal entries and the eigenvalues. For 
a triangular matrix they are the same—but that is exceptional. Normally the pivots, 
diagonal entries, and eigenvalues are completely different, And for a 2 by 2 matrix, the 
trace and determinant tell us everything: 


: ; has trace a+ d, and determinant ad — bc 
Cc 
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a-—Az b 


A—AI) = det 
det( ) =de - dod 


=e ee (trace)A + determinant 


1/2 


trace + |(trace)* — 4 det] 


pe 
Those two A’s add up to the trace; Exercise 9 gives } A; = trace for all matrices. 


The eigenvalues are A = 


Eigshow 


There is a MATLAB demo (just type eigshow), displaying the eigenvalue problem for a 
2 by 2 matrix. It starts with the unit vector x = (1,0). The mouse makes this vector move 
around the unit circle. At the same time the screen shows Ax, in color and also moving. 
Possibly Ax is ahead of x. Possibly Ax is behind x. Sometimes Ax is parallel to x. At that 
parallel moment, Ax = Ax (twice in the second figure). 


- ~ 


= LQ L- SS 
y (0, 1) A 4 ‘ S. Avy = 2} 


The eigenvalue A is the length of Ax, when the unit eigenvector x is parallel. The 
built-in choices for A illustrate three possibilities: 0, 1, or 2 real eigenvectors. 


1. There are no real eigenvectors. Ax stays behind or ahead of x. This means the 
eigenvalues and eigenvectors are complex, as they are for the rotation Q. 


2. There is only one line of eigenvectors (unusual). The moving directions Ax and x 
meet but don’t cross. This happens for the last 2 by 2 matrix below. 


3. There are eigenvectors in two independent directions. This is typical! Ax crosses x 
at the first eigenvector x;, and it crosses back at the second eigenvector x2. 


Suppose A is singular (rank 1). Its column space is a line. The vector Ax has to stay on 
that line while x circles around. One eigenvector x is along the line. Another eigenvector 
appears when Ax2 = 0. Zero is an eigenvalue of a singular matrix. 

You can mentally follow x and Ax for these six matrices. How many eigenvectors and 
where? When does Ax go clockwise, instead of counterclockwise with x? 


Paria eee 
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Problem Set 5.1 


1. 


Find the eigenvalues and eigenvectors of the matrix A = [} ]']. Verify that the trace 
equals the sum of the eigenvalues, and the determinant equals their product. 


. With the same matrix A, solve the differential equation du/dt = Au, u(0) = [?]. 


What are the two pure exponential solutions? 


If we shift to A — 7/, what are the eigenvalues and eigenvectors and how are they 


related to those of A? 
—6 -1l 
p-a-11=| 6 | 


ae aS 


Solve du/dt = Pu, when P is a projection: 


du : ) 
rs | | u with u(O) = | ; 


Part of u(0) increases exponentially while the nullspace part stays fixed. 


NIP NIE 
NIR NIE 


. Find the eigenvalues and eigenvectors of 


3. 4. 2 0 
A=1|0 12 and B= |0 
00 0 ps 


Check that A; + Az + A3 equals the trace and A, A2A3 equals the determinant. 


. Give an example to show that the eigenvalues can be changed when a multiple of one 


row is subtracted from another. Why is a zero eigenvalue not changed by the steps 
of elimination? 


. Suppose that A is an eigenvalue of A, and x is its eigenvector: Ax = Ax. 


(a) Show that this same x is an eigenvector of B = A —7/, and find the eigenvalue. 
This should confirm Exercise 3. 


(b) Assuming A # 0, show that x is also an eigenvector of A~'!—and find the eigen- 
value. 


. Show that the determinant equals the product of the eigenvalues by imagining that 


the characteristic polynomial is factored into 
det(A —AI) = (A, —A)(An—-A)---(An—-A), (16) 


and making a clever choice of A. 


o, 


10. 


11. 


12. 


13. 


14. 


15. 


16. 
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Show that the trace equals the sum of the eigenvalues, in two steps. First, find the 
coefficient of (—A)"~! on the right side of equation (16). Next, find all the terms in 


ayj—-A ayn + An 
dz, a2—-A +--+ a 
det(A—AI)=det| * id 
an\ an2d st) Ann Ar 


that involve (—A)""!. They all come from the main diagonal! Find that coefficient 
of (—A)"~! and compare. 


(a) Construct 2 by 2 matrices such that the eigenvalues of AB are not the products of 
the eigenvalues of A and B, and the eigenvalues of A + B are not the sums of the 
individual eigenvalues. 


(b) Verify, however, that the sum of the eigenvalues of A+ B equals the sum of all 
the individual eigenvalues of A and B, and similarly for products. Why is this 
true? 


The eigenvalues of A equal the eigenvalues of A‘. This is because det(A — AJ) 
equals det(A! — AJ). That is true because . Show by an example that the eigen- 
vectors of A and A! are not the same. 


Find the eigenvalues and eigenvectors of 


ab 
ba 


3 4 
4 3 


Ais 


| and A= 


If B has eigenvalues 1, 2, 3, C has eigenvalues 4, 5, 6, and D has eigenvalues 7, 8, 9, 
what are the eigenvalues of the 6 by 6 matrix A = [2 6]? 


Find the rank and all four eigenvalues for both the matrix of ones and the checker 
board matrix: 


114141 01041 
lok ft 1 1 
A= and C= 0 0 
114141 01041 
114141 1010 


Which eigenvectors correspond to nonzero eigenvalues? 


What are the rank and eigenvalues when A and C in the previous exercise are n by n? 
Remember that the eigenvalue A = 0 is repeated n — r times. 


If A is the 4 by 4 matrix of ones, find the eigenvalues and the determinant of A — /. 
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17. Choose the third row of the “companion matrix” 


0 1 0 
A=]0 0 1 


so that its characteristic polynomial |A — A/| is —A?+4A47+5A +6. 
18. Suppose A has eigenvalues 0, 3, 5 with independent eigenvectors u, v, w. 
(a) Give a basis for the nullspace and a basis for the column space. 


(b) Find a particular solution to Ax = v+ w. Find all solutions. 


(c) Show that Ax = u has no solution. (If it had a solution, then would be in 
the column space.) 


19. The powers A‘ of this matrix A approaches a limit as k — : 


4.8 Al any ah sade es: 


we at 


The matrix A? is halfway between A and A®. Explain why A” = 5(A+A®) from the 
eigenvalues and eigenvectors of these three matrices. 


20. Find the eigenvalues and the eigenvectors of these two matrices: 


1 4 2 4 
A= and A+I= : 
2.3 24 
A+T has the eigenvectors as A. Its eigenvalues are by 1. 


21. Compute the eigenvalues and eigenvectors of A and A7!: 


2, —3/4 1/2 
A= 0 and A= / / : 
2 3 1/2 0 
A~! has the eigenvectors as A. When A has eigenvalues A; and Ag, its inverse 


has eigenvalues ; 


22. Compute the eigenvalues and eigenvectors of A and A’: 


_] ee 
A= ; and A? = ; 
2 0 —2 6 
A’ has the same as A. When A has eigenvalues A; and Ay, A? has eigenvalues 


23. (a) If you know x is an eigenvector, the way to find A is to ; 


24. 


25; 


26. 


27. 


28. 


29. 


30. 
31. 
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(b) If you know A is an eigenvalue, the way to find x is to : 
What do you do to Ax = Ax, in order to prove (a), (b), and (c)? 


(a) A? is an eigenvalue of A, as in Problem 22. 
(b) A! is an eigenvalue of A~!, as in Problem 21. 
(c) A +1 is an eigenvalue of A+ J, as in Problem 20. 


From the unit vector u = (, i, ;, 2) , construct the rank-1 projection matrix P = uu!. 


(a) Show that Pu = u. Then uw is an eigenvector with A = 1. 
(b) If v is perpendicular to u show that Pv = zero vector. Then A = 0. 


(c) Find three independent eigenvectors of P all with eigenvalue A = 0. 


Solve det(Q — AJ) = 0 by the quadratic formula, to reach A = cos 9 +isin@: 


cos@ —sin@ 
sin@Q cos@ 


| rotates the xy-plane by the angle @. 
Find the eigenvectors of Q by solving (Q — AJ)x = 0. Use i? = —1. 


Every permutation matrix leaves x = (1,1,...,1) unchanged. Then A = 1. Find two 
more A’s for these permutations: 


If A has A; = 4 and Ay = 5, then det(A — AJ) = (A —4)(A —5) =A? —9A +20. Find 
three matrices that have trace a+ d = 9, determinant 20, and A = 4,5. 


A 3 by 3 matrix B is known to have eigenvalues 0, 1, 2, This information is enough 
to find three of these: 


(a) the rank of B, 

(b) the determinant of BB, 

(c) the eigenvalues of B'B, and 
(d) the eigenvalues of (B+J)~!. 


Choose the second row of A = [9 !] so that A has eigenvalues 4 and 7. 


Choose a, b, c, so that det(A — AJ) = 9A — A3. Then the eigenvalues are —3, 0, 3: 


A= 


a2 oc °° 


0 
1 
Cc 


ST OF 


272 


32. 


33% 


34. 


35. 


36. 


37. 


38. 


39. 


40. 
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Construct any 3 by 3 Markov matrix M: positive entries down each column add to 
1. If e= (1,1,1), verify that M'e =e. By Problem 11, A = 1 is also an eigenvalue 
of M. Challenge: A 3 by 3 singular Markov matrix with trace 4 has eigenvalues 


A= 


Find three 2 by 2 matrices that have A, = Az = 0. The trace is zero and the determi- 
nant is zero. The matrix A might not be 0 but check that A? = 0. 


This matrix is singular with rank 1. Find three A’s and three eigenvectors: 


1 2 2 

A=|2||2 1 2)=|4 2 4 

1 2. <2 
Suppose A and B have the same eigenvalues 11,...,A, with the same independent 
eigenvectors x1,...,X,. Then A = B. Reason: Any vector x is a combination cx; + 


-+++C,X,. What is Ax? What is Bx? 


(Review) Find the eigenvalues of A, B, and C: 


When a+ b= c-+d, show that (1,1) is an eigenvector and find both eigenvalues: 
a b 
ea 


When P exchanges rows | and 2 and columns 1 and 2, the eigenvalues don’t change. 
Find eigenvectors of A and PAP for A = 11: 


A= 


| 
6 3 and PAP = 
8 4 


oN WN 
RR WD 
Re Ww 


Challenge problem: Js there a real 2 by 2 matrix (other than J) with A> = I? Its 
eigenvalues must satisfy A> =I. They can be e2”/3 and e~27/3, What trace and 
determinant would this give? Construct A. 


There are six 3 by 3 permutation matrices P. What numbers can be the determinants 
of P? What numbers can be pivots? What numbers can be the trace of P? What four 
numbers can be eigenvalues of P? 
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5.2 Diagonalization of a Matrix 


We start right off with the one essential computation. It is perfectly simple and will be 
used in every section of this chapter. The eigenvectors diagonalize a matrix: 


5C Suppose the 1 by n matrix A has n linearly independent eigenvectors. 
If these eigenvectors are the columns of a matrix S, then S~!AS is a diagonal 
matrix A. The eigenvalues of A are on the diagonal of A: 
Ay 
3 3 3 —1 Az 
Diagonalization SAS =A= ‘ (1) 


An 


> 


We call S the “eigenvector matrix” and A the “eigenvalue matrix”—using a capital 
lambda because of the small lambdas for the eigenvalues on its diagonal. 


Proof. Put the eigenvectors x; in the columns of S, and compute AS by columns: 


AS=A lx, Xo. +++ Xl = |Apxy Aox2 «++ AnXn 
| | | | | | 
Then the trick is to split this last matrix into a quite different product SA: 
A 
Ay 
Aix, Aoxoa +++ AnXy| = |X] X20 + Xp 


An 


It is crucial to keep these matrices in the right order. If A came before S$ (instead of 
after), then A; would multiply the entries in the first row. We want A; to appear in the 
first column. As it is, SA is correct. Therefore, 


AS=SA, or S-'AS=A, or A=SAS™!. (2) 


S is invertible, because its columns (the eigenvectors) were assumed to be independent. 
We add four remarks before giving any examples or applications. 


Remark 1. If the matrix A has no repeated eigenvalues—the numbers A),...,A, are 
distinct—then its n eigenvectors are automatically independent (see 5D below). There- 
fore any matrix with distinct eigenvalues can be diagonalized. 


Remark 2. The diagonalizing matrix S is not unique. An eigenvector x can be multiplied 
by a constant, and remains an eigenvector. We can multiply the columns of S by any 
nonzero constants, and produce a new diagonalizing S. Repeated eigenvalues leave even 
more freedom in S. For the trivial example A = /, any invertible S will do: S~'JS is is 
always diagonal (A is just J). All vectors are eigenvectors of the identity. 
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Remark 3. Other matrices S will not produce a diagonal A. Suppose the first column 
of S is y. Then the first column of SA is A,y. If this is to agree with the first column of 
AS, which by matrix multiplication is Ay, then y must be an eigenvector: Ay = A, y. The 
order of the eigenvectors in S and the eigenvalues in A is automatically the same. 


Remark 4, Not all matrices possess n linearly independent eigenvectors, so not all ma- 
trices are diagonalizable. The standard example of a “defective matrix” is 


Its eigenvalues are A; = Az = 0, since it is triangular with zeros on the diagonal: 


—-r 1 ) 
det(A — AJ) = det | 0 a Xr 


All eigenvectors of this A are multiples of the vector (1,0): 


A =0 is a double eigenvalue—its algebraic multiplicity is 2. But the geometric multi- 
plicity is 1—there is only one independent eigenvector. We can’t construct S. 

Here is a more direct proof that this A is not diagonalizable. Since A; = Az = 0, A 
would have to be the zero matrix, But if A = S~'AS = 0, then we premultiply by S and 
postmultiply by S~!, to deduce falsely that A = 0. There is no invertible S. 

That failure of diagonalization was not a result of A = 0. It came from A; = Az: 


2 -1 

1 0} 

Their eigenvalues are 3, 3 and 1, 1. They are not singular! The problem is the shortage 
of eigenvectors—which are needed for S. That needs to be emphasized: 


Repeated eigenvalues A= 


ae and A= 
0 3 


Diagonalizability of A depends on enough eigenvectors. 
Invertibility of A depends on nonzero eigenvalues. 


There is no connection between diagonalizability (n independent eigenvector) and in- 
vertibility (no zero eigenvalues). The only indication given by the eigenvalues is this: 
Diagonalization can fail only if there are repeated eigenvalues. Even then, it does not 
always fail. A =/ has repeated eigenvalues 1,1,...,1 but it is already diagonal! There 
is no shortage of eigenvectors in that case. 

The test is to check, for an eigenvalue that is repeated p times, whether there are p 
independent eigenvectors—in other words, whether A — AJ has rank n — p. To complete 
that circle of ideas, we have to show that distinct eigenvalues present no problem. 


5.2 Diagonalization of a Matrix 275 


5D If eigenvectors x1,...,x, correspond to different eigenvalues 1,,...,Ax, 
then those eigenvectors are linearly independent. 


Suppose first that k = 2, and that some combination of x; and x2 produces zero: 
C1xX1 +¢C2x2 = 0. Multiplying by A, we find cjA,x1; + c2A2x2 = 0. Subtracting A times 
the previous equation, the vector x2 disappears: 


c(Ay — Az)x; = 0. 


Since A, 4 Ap and x; # 0, we are forced into c} = 0. Similarly cp = 0, and the two 
vectors are independent; only the trivial combination gives zero. 

This same argument extends to any number of eigenvectors: If some combination pro- 
duces zero, multiply by A, subtract A, times the original combination, and x; disappears— 
leaving a combination of x;,...,x,_1, which produces zero. By repeating the same steps 
(this is really mathematical induction) we end up with a multiple of x; that produces 
zero. This forces cj = 0, and ultimately every c; = 0. Therefore eigenvectors that come 
from distinct eigenvalues are automatically independent. 

A matrix with n distinct eigenvalues can be diagonalized. This is the typical case. 


Examples of Diagonalization 


The main point of this section is S~!'AS = A. The eigenvector matrix S converts A into 
its eigenvalue matrix A (diagonal). We see this for projections and rotations. 


4 
Example 1. The projection A = i has eigenvalue matrix A = [} }]. The eigen- 
22 


vectors go into the columns of S: 


1 0 


0 


s= ; | and AS = SA = 


That last equation can be verified at a glance. Therefore S~!AS = A. 


Example 2. The eigenvalues themselves are not so clear for a rotation: 


0 
1 


90° rotation K= 


‘i has det(K—Al) =A7+1. 


How can a vector be rotated and still have its direction unchanged? Apparently it 
can’t—except for the zero vector, which is useless. But there must be eigenvalues, and 
we must be able to solve du/dt = Ku. The characteristic polynomial A? + 1 should still 
have two roots—but those roots are not real. 


You see the way out. The eigenvalues of K are imaginary numbers, A, =i and Az = 
—i. The eigenvectors are also not real. Somehow, in turning through 90°, they are 
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multiplied by 7 or —i: 


cowl fl  = =( 
come EE) = fh 


The eigenvalues are distinct, even if imaginary, and the eigenvectors are independent. 
They go into the columns of S: 

i 0 

0 —-i| 


We are faced with an inescapable fact, that complex numbers are needed even for 
real matrices. If there are too few real eigenvalues, there are always n complex eigen- 
values. (Complex includes real, when the imaginary part is zero.) If there are too few 
eigenvectors in the real world R?, or in R”, we look in C? or C”. The space C” contains 
all column vectors with complex components, and it has new definitions of length and 
inner product and orthogonality. But it is not more difficult than R”, and in Section 5.5 
we make an easy conversion to the complex case. 


—l oi 


ig 3 
s-| and = S-'KS = 


Powers and Products: A‘ and AB 


There is one more situation in which the calculations are easy. The eigenvalue of A? are 
exactly Az ...,A2, and every eigenvector of A is also an eigenvector of A*. We start 
from Ax = Ax, and multiply again by A: 


A’x =AAx = AAx = A?x. (3) 


Thus A? is an eigenvalue of A*, with the same eigenvector x. If the first multiplication 
by A leaves the direction of x unchanged, then so does the second. 
The same result comes from diagonalization, by squaring S~'AS = A: 


Eigenvalues of A” (S-'as)(S“!AS) =A? or: S7'A?S= A’. 


The matrix A” is diagonalized by the same S, so the eigenvectors are unchanged. The 
eigenvalues are squared. This continues to hold for any power of A: 


5E The eigenvalues of A‘ are Aes ..., A, and each eigenvector of A is still an 
eigenvector of A‘. When S diagonalizes A, it also diagonalizes A’: 


AF = (S-!AS)(S“!AS) --- (S~1AS) = S“1AKS. (4) 


Each S~! cancels an S, except for the first S~! and the last S. 
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If A is invertible this rule also applies to its inverse (the power k = —1). The eigen- 
values of A~' are 1/A;. That can be seen even without diagonalizing: 


1 
if Ax=Ax then x=AA™!'x and qi Ax. 


Example 3. If K is rotation through 90°, then K? is rotation through 180° (which means 
—I) and K~! is rotation through —90°: 


—] —1 1 
: ‘ K? = ss , and K — : 
1 0 0 -l —1 0 
The eigenvalues of K are i and —i; their squares are —1 and —1; their reciprocals are 
1 /i= —iand 1/(—i) =i. Then K* is a complete rotation through 360°: 


*# oO | |1 0 
0 (-i*| |0 1 


For a product of two matrices, we can ask about the eigenvalues of AB—but we won’t 
get a good answer. It is very tempting to try the same reasoning, hoping to prove what 
is not in general true. If A is an eigenvalue of A and yp is an eigenvalue of B, here is the 
false proof that AB has the eigenvalue UA: 


K= 


K‘= 


0 1 


1 
andalso A* = 


False proof ABx=Apmx = UWAx = Ax. 


The mistake lies in assuming that A and B share the same eigenvector x. In general, they 
do not, We could have two matrices with zero eigenvalues, while AB has A = 1: 


sIpd-b3 


The eigenvectors of this A and B are completely different, which is typical. For the same 
reason, the eigenvalues of A+B generally have nothing to do with A + p. 

This false proof does suggest what is true. If the eigenvector is the same for A and 
B, then the eigenvalues multiply and AB has the eigenvalue A. But there is something 
more important. There is an easy way to recognize when A and B share a full set of 
eigenvectors, and that is a key question in quantum mechanics: 


AB= 


5F Diagonalizable matrices share the same eigenvector matrix S if and only 
if AB = BA. 


Proof. If the same S diagonalizes both A = SA,S~! and B = SA2S~!, we can multiply 
in either order: 


AB = SA,S~!SA>S~! = SA,A,S~! and BA =SA,S~!SA,S~! = SAvA,S7!. 


Since A; Ay = A2A, (diagonal matrices always commute) we have AB = BA. 
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In the opposite direction, suppose AB = BA. Starting from Ax = Ax, we have 


ABx = BAx = BAx = ABx. 


Thus x and Bx are both eigenvectors of A, sharing the same A (or else Bx = 0). If we 
assume for convenience that the eigenvalues of A are distinct—the eigenspaces are all 
one-dimensional—then Bx must be a multiple of x. in other words x is an eigenvector of 
B as well as A. The proof with repeated eigenvalues is a little longer. 


Heisenberg’s uncertainty principle comes from noncommuting matrices, like posi- 
tion P and momentum Q. Position is symmetric, momentum is skew-symmetric, and 
together they satisfy QP — PQ =I. The uncertainty principle follows directly from the 
Schwarz inequality (Qx)!(Px) < ||Qx||||Px|| of Section 3.2: 


||x||? = xt x = x™(QP — PQ)x < 2||Ox||||Px||. 


The product of ||Q-x||/||x|| and ||P.x||/||x||—momentum and position errors, when the 
wave function is x—is at least 5. It is impossible to get both errors small, because when 
you try to measure the position of a particle you change its momentum. 

At the end we come back to A = SAS~!. That factorization is particularly suited to 
take powers of A, and the simplest case A* makes the point. The LU factorization is 
hopeless when squared, but SAS~! is perfect. The square is SA*S~', and the eigenvec- 
tors are unchanged. By following those eigenvectors we will solve difference equations 
and differential equations. 


Problem Set 5.2 


1. Factor the following matrices into SAS~!: 


1 1 
1 1 


A= 


| and A= 


ame | 

0 O}- 

2. Find the matrix A whose eigenvalues are 1 and 4, and whose eigenvectors are [?] 
and [+], respectively. (Hint: A= SAS~'.) 


3. Find all the eigenvalues and eigenvectors of 


ee eee 
pm pk fk 


and write two different diagonalizing matrices S. 


4. If a3 by 3 upper triangular matrix has diagonal entries 1, 2, 7, how do you know it 
can be diagonalized? What is A? 


10. 


11. 


12. 


13. 


14. 
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Which of these matrices cannot be diagonalized? 


2S 22 
22 


2 O 


Ao = 
BSN: 255 


(a) If A* = J, what are the possible eigenvalues of A? 

(b) If this A is 2 by 2, and not J or —/, find its trace and determinant. 
(c) If the first row is (3,—1), what is the second row? 

If A = [43], find A!” by diagonalizing A. 


T 


Suppose A = uv" is acolumn times a row (a rank-1 matrix). 


(a) By multiplying A times wu, show that u is an eigenvector. What is 2? 
(b) What are the other eigenvalues of A (and why)? 


(c) Compute trace(A) from the sum on the diagonal and the sum of 1’s. 


Show by direct calculation that AB and BA have the same trace when 


- and = | i; 
cd s ¢t 


Deduce that AB — BA = / is impossible (except in infinite dimensions). 


A= 


Suppose A has eigenvalues 1, 2, 4. What is the trace of A*? What is the determinant 
of (A=)? 


If the eigenvalues of A are 1, 1, 2, which of the following are certain to be true? Give 
a reason if true or a counterexample if false: 

(a) A is invertible. 

(b) A is diagonalizable. 


(c) A is not diagonalizable. 
Suppose the only eigenvectors of A are multiples of x = (1,0,0). True or false: 


(a) A is not invertible. 
(b) A has a repeated eigenvalue. 
(c) A is not diagonalizable. 


Diagonalize the matrix A = [3 ¢] and find one of its square roots—a matrix such that 
R? = A. How many square roots will there be? 


Suppose the eigenvector matrix S has S' = S—!. Show that A = SAS~! is symmetric 
and has orthogonal eigenvectors. 


Problems 15-24 are about the eigenvalue and eigenvector matrices. 
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15. 


16. 
17. 


18. 


19. 


20. 


21. 


22. 
23. 


24. 
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Factor these two matrices into A = SAS7!: 


L'2 
0 3 


Ac 


| and A= 


11 
2.2\, 
IfA=SAS"'thenA?>=( )(  )( )andAt=( /)C /) )z 


If A has A, = 2 with eigenvector x; = |] and Az =5 with x2 = [1], use SAS! to 
find A. No other matrix has the same A’s and x’s. 


Suppose A = SAS~!. What is the eigenvalue matrix for A+ 27? What is the eigen- 
vector matrix? Check that A + 27 = ( )( )( 4. 


True or false: If the n columns of S (eigenvectors of A) are independent, then 


(a) A is invertible. 
(b) A is diagonalizable. 
(c) § is invertible. 


(d) Sis diagonalizable. 


If the eigenvectors of A are the columns of /, then A is a matrix. If the eigen- 
vector matrix S is triangular, then S~! is triangular and A is triangular. 


Describe all matrices S that diagonalize this matrix A: 
4 0 
1 21° 


Then describe all matrices that diagonalize A~!. 


ae 


Write the most general matrix that has eigenvectors | || and [| |, ]. 


Find the eigenvalues of A and B and A +B: 


1 


1 1 0 1 


1 1 
| A+B= 


Eigenvalues of A+ B (are equal to)(are not equal to) eigenvalues of A plus eigenval- 
Find the eigenvalues of A, B, AB, and BA: 


ues of B. 
1 1 2. st 
, B= , AB= 
0 1 1 1 


Eigenvalues of AB (are equal to)(are not equal to) eigenvalues of A times eigenvalues 
of B. Eigenvalues of AB (are)(are not) equal to eigenvalues of BA. 


1 0 
1 1 


A= 
1 2 


1 1 
| and BA= 
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Problems 25-28 are about the diagonalizability of A. 
25. True or false: If the eigenvalues of A are 2, 2, 5, then the matrix is certainly 
(a) invertible. 
(b) diagonalizable. 
(c) not diagonalizable. 


26. If the eigenvalues of A are | and 0, write everything you know about the matrices A 
and A’. 


27. Complete these matrices so that detA = 25. Then trace = 10, and A = 5 is repeated! 
Find an eigcnvector with Ax = 5x. These matrices will nothe diagonalizabie because 
there is no second line of eigenvectors. 

10 5 
235 ; 


28. The matrix A = [34] is not diagonalizable because the rank of A — 3/ is 
Change one entry to make A diagonalizable. Which entries could you change? 


8 
Z 


A= , A= 


4 
A iF and A= 


Problems 29-33 are about powers of matrices. 


29. A‘ = SA‘S~! approaches the zero matrix as k — © if and only if every A has absolute 
value less than . Does A‘ — 0 or BK > 0? 


6 4 6 9 
B= . 
4 ue E q 
30. (Recommended) Find A and S to diagonalize A in Problem 29. What is the limit of 


AF as k — co? What is the limit of SA‘S~!? In the columns of this limiting matrix 
you see the : 


a 


31. Find A and S to diagonalize B in Problem 29. What is B'°uo for these ug? 


geet meee ae desl? 
o=]a]° e412 0= Jol: 


32. Diagonalize A and compute SA‘S~! to prove this formula for A*: 


3K41 3k-] 
3K 1 3k 4] 


2 
| ee 


1 


A= | has Ase 


pa 


33. Diagonalize B and compute SA‘S~! to prove this formula for B*: 


1 k k _9k 
ie =) aa ; . 
0 2 0 2 


has BS | 
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Problems 34—44 are new applications of A = SAS™!. 


34. 


35. 


36. 


Shs 


38. 


39. 


40. 


41. 


42. 


43. 


44. 


45. 


Suppose that A= SAS~!. Take determinants to prove that detA = A,A2--- An = prod- 
uct of A’s. This quick proof only works when A is ‘ 


The trace of S times AS~! equals the trace of AS~! times S. So the trace of a diago- 
nalizable A equals the trace of A, which is : 


If A = SAS™', diagonalize the block matrix B = [4 9,|. Find its eigenvalue and 
eigenvector matrices. 


Consider all 4 by 4 matrices A that are diagonalized by the same fixed eigenvector 
matrix $. Show that the A’s form a subspace (cA and A; +A? have this same S). 
What is this subspace when S$ = J? What is its dimension? 


Suppose A? = A. On the left side A multiplies each column of A. Which of our four 
subspaces contains eigenvectors with A = 1? Which subspace contains eigenvectors 
with A = 0? From the dimensions of those subspaces, A has a full set of independent 
eigenvectors and can be diagonalized. 


Suppose Ax = Ax. If A =0, then x is in the nullspace. If A 4 0, then x is in the 
column space. Those spaces have dimensions (n — r) +r =n. So why doesn’t every 
square matrix have n linearly independent eigenvectors? 


Substitute A = SAS“! into the product (A — A,/)(A — Agl)---(A — Ant) and explain 
why this produces the zero matrix. We are substituting the matrix A for the number 


A in the polynomial p(A) = det(A — AJ). The Cayley-Hamilton Theorem says that 
this product is always p(A) = zero matrix, even if A is not diagonalizable. 


Test the Cayley-Hamilton Theorem on Fibonacci’s matrix A = [| 4]. The theorem 
predicts that A7 A —I =O, since det(A—AJ) isA7—A—1. 

If A = [44], then det(A — AJ) is (A —a)(A —d). Check the Cayley-Hamilton state- 
ment that (A — al)(A — dI) = zero matrix. 

If A = [49] and AB = BA, show that B = [44] is also diagonal. B has the same 
eigen as A, but different eigen . These diagonal matrices B form a two- 


dimensional subspace of matrix space. AB — BA = 0 gives four equations for the 
unknowns a, b, c, d—find the rank of the 4 by 4 matrix. 


If A is 5 by 5. then AB— BA = zero matrix gives 25 equations for the 25 entries in B. 
Show that the 25 by 25 matrix is singular by noticing a simple nonzero solution B. 


Find the eigenvalues and eigenvectors for both of these Markov matrices A and A”. 
Explain why A! is close to A®: 


6 H air iiss Be ver 


Ae 
A 8 2/3 2/3 
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5.3 Difference Equations and Powers A“ 


Difference equations uz1; = Au, move forward in a finite number of finite steps. A 
differential equation takes an infinite number of infinitesimal steps, but the two theories 
stay absolutely in parallel. It is the same analogy between the discrete and the continuous 
that appears over and over in mathematics. A good illustration is compound interest, 
when the time step gets shorter. 

Suppose you invest $1000 at 6% interest. Compounded once a year, the principal P 
is multiplied by 1.06. This is a difference equation P,., = AP, = 1.06P, with a time step 
of one year. After 5 years, the original Py = 1000 has been multiplied 5 times: 


Yearly Ps =(1.06)°Py whichis (1.06)°1000 = $1338. 


Now suppose the time step is reduced to a month. The new difference equation is py, = 
(1+ .06/12)p,. After 5 years, or 60 months, you have $11 more: 


06\ °° 
Monthly — p69 = (: us — po whichis (1.005)°°1000 = $1349. 
The next step is to compound every day, on 5(365) days. This only helps a little: 


365 


Finally, to keep their employees really moving, banks offer continuous compounding. 
The interest is added on at every instant, and the difference equation breaks down. You 
can hope that the treasurer does not know calculus (which is all about limits as At — 0). 
The bank could compound the interest N times a year, so At = 1/N: 


06 \ 5365 
Daily compounding (1 + a) 1000 = $1349.83. 


06\ >" 
Continuously (1 ap + 1000 — e°°1000 = $1349.87. 
Or the bank can switch to a differential equation—the limit of the difference equation 
Peoi = (14+ .06Ar) pz. Moving p; to the left side and dividing by Ar, 


Di te t > d 
ete eto Pietl PR .06p, approaches <P = -O6p. (1) 
continuous At at 


The solution is p(t) = e°” po. After t = 5 years, this again amounts to $1349.87. The 
principal stays finite, even when it is compounded every instant—and the improvement 
over compounding every day is only four cents. 


Fibonacci Numbers 


The main object of this section is to solve uz4; = Aux. That leads us to A* and powers 
of matrices. Our second example is the famous Fibonacci sequence: 


Fibonacci numbers he) PA Its Fao Pe. ie ho eee 
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You see the pattern: Every Fibonacci number is the sum of the two previous F’s: 
Fibonacci equation Fuso = Fea tly. (2) 


That is the difference equation. It turns up in a most fantastic variety of applications, 
and deserves a book of its own. Leaves grow in a spiral pattern, and on the apple or oak 
you find five growths for every two turns around the stem. The pear tree has eight for 
every three turns, and the willow is 13:5. The champion seems to be a sunflower whose 
seeds chose an almost unbelievable ratio of F,2/F\3 = 144/233.” 

How could we find the 1000th Fibonacci number, without starting at Fy = 0 and 
F, = 1, and working all the way out to Fjg99? The goal is to solve the difference equation 
Fus2 = Fy41+Fy. This can be reduced to a one-step equation u;,4; = Au;,. Every step 
multiplies u, = (F,+41,F,) by a matrix A: 


Fyyo = Papa t+ Pe 
Prey = Pr 


1 1 
1 0 


Fry 
Fy 


= Aug. (3) 


becomes U4] = | 


The one-step system uz41; = Aug is easy to solve, It starts from uo. After one step it 
produces uj = Aug. Then wz is Auy, which is Aug. Every step brings a multiplication 
by A, and after k steps there are k multiplications: 


The solution to a difference equation uz., = Au, is uz, = Akuo. 


The real problem is to find some quick way to compute the powers A‘, and thereby find 
the 1000th Fibonacci number. The key lies in the eigenvalues and eigenvectors: 


5G_ If A can be diagonalized, A= SAS~!, then A‘ comes from A‘: 


uz = Akug = (SAS~!)(SAS~!)---(SAS~!)up = SAKS~!up. (4) 
The columns of S are the eigenvectors of A. Writing S~'uo = c, the solution 
becomes 
A‘ Cl 
uz = SA*e = Xi °*: Xp Wee : = cyAby +++ ten Aenn. 
AE Nea 


After k steps, uz is a combination of the n “pure solutions” A‘x. 
These formulas give two different approaches to the same solution uz = SA‘S~!uo. 
The first formula recognized that A‘ is identical with SA‘S~', and we could stop there. 


>For these botanical applications, see D’ Arcy Thompson’s book On Growth and Form (Cambridge University 
Press, 1942) or Peter Stevens’s beautiful Patterns in Nature (Little, Brown, 1974). Hundreds of other properties 
of the F;, have been published in the Fibonacci Quarterly. Apparently Fibonacci brought Arabic numerals into 
Europe, about 1200 A.D. 
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But the second approach brings out the analogy with a differential equation: The pure 
exponential solutions e”'x; are now the pure powers ARx;. The eigenvectors x; are 
amplified by the eigenvalues A;. By combining these special solutions to match uwy—that 
is where c came from—we recover the correct solution u, = SA‘S~!ug. 

In any specific example like Fibonacci’s, the first step is to find the eigenvalues: 


A-Al= i a | has det(A—AI)=A?-A-1 
Two eigenvalues A, = : We and A, = : — 


The second row of A— AJ is (1,—A). To get (A—A/)x = 0, the eigenvector is x = (A, 1), 
The first Fibonacci numbers Fo = 0 and F; = 1 go into uo, and S-tug =c: 


af 
Fe Ay Ao 1 : 1/(Ay — Az) 1 1 
Clg = ives c= == : 
E | ; . lel aes est 
Those are the constants in u,z = cA ~ coAK XD. Both eigenvectors x; and x2 have 
second component |. That leaves F; = cA _ coAk in the second component of ux: 


k k 
Fibonacci =, (54) (54) 


numbers J5 2 2 


This is the answer we wanted. The fractions and square roots look surprising because 
Fibonacci’s rule Fx42 = Fy, + F, must produce whole numbers, Somehow that formula 
for F;, must give an integer. In fact, since the second term [(1 — 5) /2]*/\/5 is always 
less than > it must just move the first term to the nearest integer: 


1000 
1 ( 1475 
F090 = nearest integer to —~ 


We. 2 


This is an enormous number, and Fjg9; will be even bigger. The fractions are becoming 
insignificant, and the ratio F901 /F,o00 must be very close to (1 + V/5)/2 = 1.618. Since 
AX is insignificant compared to A‘, the ratio Fy. /F; approaches A). 

That is a typical difference equation, leading to the powers of A = E me it involved 
\/5 because the eigenvalues did. If we choose a matrix with A; = 1 and A, = 6. we can 
focus on the simplicity of the computation—after A has been diagonalized: 


4 _ 1 = 
A= | : has A=land6, with x,= | and x= FA 
te etl) ae Oe Oe 2— 6% 1—6* 
Ak=SAKS"! i = 
= le | F : ; i 242-6 —142-6é 
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The powers 6* and 1* appear in that last matrix A‘, mixed in by the eigenvectors. 
For the difference equation uz; = Aux, we emphasize the main point. Every eigen- 
vector x produces a “pure solution” with powers of 1: 


One solution is ug=xX, w=Ax, w=A2n,... 


When the initial wo is an eigenvector x, this is the solution: uz = A*x. In general ug 
is not an eigenvector. But if uo is a combination of eigenvectors, the solution uz is the 
same combination of these special solutions. 


5H If wy = cyx1 +--+ +CpXp, then after k steps uz = cyAkxy Spee sic Cd. 
Choose the c’s to match the starting vector uo: 


Up = |X, +++ Xp >| = Sc and c=S luo. (6) 


Markov Matrices 


There was an exercise in Chapter 1, about moving in and out of California, that is worth 
another look. These were the rules: 


Each year ib of the people outside California move in, and i of the people 
inside California move out. We start with yg people outside and Zo inside. 


At the end of the first year the numbers outside and inside are y; and 2: 


= 9yo + .2z9 ee. WEN 9 21 lyo 
= .lyo + .820 ZI 1.8] | zo] 


This problem and its matrix have the two essential properties of a Markov process: 


Difference y 


ay 


equation Zz 


= 


1. The total number of people stays fixed: Each column of the Markov matrix adds 
up to 1. Nobody is gained or lost. 


2. The numbers outside and inside can never become negative: The matrix has no 
negative entries. The powers A* are all nonnegative.* 


We solve this Markov difference equation using uz = SA‘S~!ug. Then we show that 
the population approaches a “steady state.” First A has to be diagonalized: 


9-A 2 


a ee | 1 8A 


| has det(A—AI) =A?—1.7A+.7 


3Furthermore, history is completely disregarded; each new u,1, depends only on the current u;,. Perhaps even 
our lives are examples of Markov processes, but I hope not. 
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i — 4k [0] i ‘| fi ; I e 
Zk 20 5-4 TEV NA 28) ezG 
2 1 
= (yo +20) |} + (0 — 220)(.7)* 4. 
3 3 


Those two terms are cA ~ coAKx. The factor Ae = | is hidden in the first term. In 
the long run, the other factor (.7)* becomes extremely small. The solution approaches 
a limiting state Us. = (Yoo, Z00): 


The total population is still yo + zo, but in the limit $ of this population is outside Cali- 
fornia and i is inside. This is true no matter what the initial distribution may have been! 
If the year starts with 5 outside and ; inside, then it ends the same way: 


nll 


The steady state is the eigenvector of A corresponding to 1 = 1. Multiplication by A, 
from one time step to the next, leaves u.. unchanged. 
The theory of Markov processes is illustrated by that California example: 


foe} 


Steady state P = (yo +20) 
z 


WI QW1N 


2 
| or Allso = Uso. 
3 


SI A Markov matrix A has all aj; > 0, with each column adding to 1. 


(a) A; = 1 is an eigenvalue of A. 

(b) Its eigenvector x; is nonnegative—and it is a steady state, since Ax; = x). 
(c) The other eigenvalues satisfy ||A;|| < 1. 

(d) If A or any power of A has all positive entries, these other |A;| are below 1. 


The solution A‘ug approaches a multiple of xj—which is the steady state 
Yeni 


To find the right multiple of x;, use the fact that the total population stays the same. If 
California started with all 90 million people out, it ended with 60 million out and 30 
million in. It ends the same way if all 90 million were originally inside. 

We note that many authors transpose the matrix so its rows add to 1. 


Remark. Our description of a Markov process was deterministic: populations moved in 
fixed proportions. But if we look at a single individual, the fractions that move become 
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probabilities. With probability a an individual outside California moves in. If inside, 
the probability of moving out is a The movement becomes a random process, and A is 
called a transition matrix. 

The components of u; = Aug specify the probability that the individual is outside 
or inside the state. These probabilities are never negative and add to 1—everybody has 
to be somewhere. That brings us back to the two fundamental properties of a Markov 
matrix: Each column adds to 1, and no entry is negative. 


Why is A = | always an eigenvalue? Each column of A —/ adds up to 1—1=0. 
Therefore the rows of A —J add up to the zero row, they are linearly dependent, and 
det(A —/) = 0. 

Except for very special cases, u; will approach the corresponding eigenvector*. In 
the formula uz = cyAkxy free Codey, no eigenvalue can be larger than 1. (Otherwise 
the probabilities uz, would blow up.) If all other eigenvalues are strictly smaller than 
A; = 1, then the first term in the formula will be dominant. The other A go to zero, and 
Ux — C{X| = Uo = Steady state. 

This is an example of one of the central themes of this chapter: Given information 
about A, find information about its eigenvalues. Here we found Amax = 1. 


Stability of Up+| = Au 


There is an obvious difference between Fibonacci numbers and Markov processes. The 
numbers F; become larger and larger, while by definition any “probability” is between 0 
and 1. The Fibonacci equation is unstable. So is the compound interest equation P,., = 
1.06P;,; the principal keeps growing forever. If the Markov probabilities decreased to 
zero, that equation would be stable; but they do not, since at every stage they must add 
to 1. Therefore a Markov process is neutrally stable. 

We want to study the behavior of uz; = Aux as k — oc. Assuming that A can be 
diagonalized, u; will be a combination of pure solutions: 


Solution at time k “= SAKS— lug = cyAkxy feeet CMe xn. 
The growth of uz is governed by the A. Stability depends on the eigenvalues: 
5J The difference equation uz.) = Au, is 


stable if all eigenvalues satisfy |A;| < 1; 
neutrally stable if some |A;| = 1 and all the other |A;| < 1; and 
unstable if at least one eigenvalue has |A;| > 1. 


In the stable case, the powers A* approach zero and so does uz = A*uo. 


4If everybody outside moves in and everybody inside moves out, then the populations are reversed every year 
and there is no steady state. The transition matrix is A = iy Al and —1 is an eigenvalue as well as +1—which 
cannot happen if all a;; > 0. 
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Example 1. This matrix A is certainly stable: 


0 4 


A= 
0 5 


f 
| has eigenvalues 0 and > 


The A’s are on the main diagonal because A is triangular. Starting from any uo, and 
following the rule uz. = Aux, the solution must eventually approach zero: 


uo = > w= {1/, wali, Ball, 4=HlTlo-: 
J 2 4 8 16 


The larger eigenvalue A = 4 governs the decay; after the first step every uz is Sug}. The 
real effect of the first step is to split uo into the two eigenvectors of A: 
—8 
Pale 


1\*{8 
and then uz = (5) | | + (0) 
2 1 
Positive Matrices and Applications in Economics 

By developing the Markov ideas we can find a small gold mine (entirely optional) of 
matrix applications in economics. 


ug = 


1 0 


Example 2 (Leontief’s input-output matrix). 

This is one of the first great successes of mathematical economics. To illustrate it, we 
construct a consumption matrix—in which aj;;, gives the amount of product j that is 
needed to create one unit of product i: 


4 0 .1 (steel) 
A= 0-2 38) 4 (food) 
Sade al (labor) 
The first question is: Can we produce y units of steel, y2 units of food, and y3 units of 
labor? We must start with larger amounts p;, p2, p3, because some part is consumed 
by the production itself. The amount consumed is Ap, and it leaves a net production of 
p—Ap. 
Problem To find a vector p such that p—Ap=y, or p=(I—A) ly. 


On the surface, we are only asking if J — A is invertible. But there is a nonnegative twist 
to the problem. Demand and production, y and p, are nonnegative. Since p is (1 —A)~'y, 
the real question is about the matrix that multiplies y: 


When is (I —A)~' a nonnegative matrix? 


Roughly speaking, A cannot be too large. If production consumes too much, nothing is 
left as output. The key is in the largest eigenvalue A, of A, which must be below 1: 
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If A; > 1, (I—A)7! fails to be nonnegative. 
If A; = 1, (1—A)~! fails to exist. 
If A; <1, (I—A)~! is a converging sum of nonnegative matrices: 


Geometric series (I-A) "| =I+A+A*+A34---. (7) 


The 3 by 3 example has A; = .9, and output exceeds input. Production can go on. 

Those are easy to prove, once we know the main fact about a nonnegative matrix like 
A: Not only is the largest eigenvalue A, positive, but so is the eigenvector x,. Then 
(I —A)~! has the same eigenvector, with eigenvalue 1/(1 — A). 

If A; exceeds 1, that last number is negative. The matrix ([—A)~! will take the 
positive vector x; to a negative vector x,/(1— Aj). In that case (— A)~! is definitely 
not nonnegative. If A; = 1, then J —A is singular. The productive case is A; < 1, when 
the powers of A go to zero (stability) and the infinite series ]-+A+A7+--- converges. 
Multiplying this series by J — A leaves the identity matrix—all higher powers cancel—so 
(I—A)~! is a sum of nonnegative matrices, We give two examples: 


0 2 
A= 5 ‘| has A; = 2 and the economy is lost 

ao 2 1 : 
A= 0 5 has A; = 5 and we can produce anything. 


The matrices ([— A)! in those two cases are —; [47] and [2 8]. 


Leontief’s inspiration was to find a model that uses genuine data from the real econ- 
omy. The table for 1958 contained 83 industries in the United States, with a “trans- 
actions table” of consumption and production for each one. The theory also reaches 
beyond (J — A)~!, to decide natural prices and questions of optimization. Normally la- 
bor is in limited supply and ought to be minimized. And, of course, the economy is not 
always linear. 


Example 3 (The prices in a closed input-output model! ). 

The model is called “closed” when everything produced is also consumed. Nothing goes 
outside the system. In that case A goes back to a Markov matrix. The columns add up 
to 1. We might be talking about the value of steel and food and labor, instead of the 
number of units, The vector p represents prices instead of production levels. 

Suppose po is a vector of prices. Then Apo multiplies prices by amounts to give the 
value of each product. That is a new set of prices which the system uses for the next 
set of values Apo. The question is whether the prices approach equilibrium. Are there 
prices such that p = Ap, and does the system take us there? 

You recognize p as the (nonnegative) eigenvector of the Markov matrix A, with A = 1. 
It is the steady state p.., and it is approached from any starting point po. By repeating a 
transaction over and over, the price tends to equilibrium. 
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The “Perron-Frobenius theorem” gives the key properties of a positive matrix—not 
to be confused with a positive definite matrix, which is symmetric and has all its eigen- 
values positive. Here all the entries a;; are positive. 


5K_ If A is a positive matrix, so is its largest eigenvalue: A; > all other |A/|. 
Every component of the corresponding eigenvector x; is also positive. 


Proof. Suppose A > 0. The key idea is to look at all numbers ¢ such that Ax > tx for 
some nonnegative vector x (other than x = 0). We are allowing inequality in Ax > tx in 
order to have many positive candidates t. For the largest value tmax (which is attained), 
we will show that equality holds: Ax = tmaxx. 

Otherwise, if Ax > tmaxx 1s not an equality, multiply by A. Because A is positive, that 
produces a strict inequality A2x > tmaxAx. Therefore the positive vector y = Ax satisfies 
Ay > tmaxy, and fmax could have been larger. This contradiction forces the equality Ax = 
tmaxX, and we have an eigenvalue. Its eigenvector x is positive (not just nonnegative) 
because on the left-hand side of that equality Ax is sure to be positive. 

To see that no eigenvalue can be larger than tmax, suppose Az = Az. Since A and z 
may involve negative or complex numbers, we take absolute values: |A||z| = |Az| < Alz| 
by the “triangle inequality.” This |z| is a nonnegative vector, so |A| is one of the possible 
candidates t. Therefore |A| cannot exceed A, which was fax. 


Example 4 (Von Neumann’s model of an expanding economy ). 
We go back to the 3 by 3 matrix A that gave the consumption of steel, food, and labor. 
If the outputs are s1, f;, 21, then the required inputs are 


4 0 .1] | sy 
u=|0 .1 .8] | fi, | =Aut. 
iia Gare 2) 


In economics the difference equation is backward! Instead of uv; = Aug we have uo = 
Au. If A is small (as it is), then production does not consume everything—and the 
economy can grow. The eigenvalues of A~! will govern this growth. But again there is 
a nonnegative twist, since steel, food, and labor cannot come in negative amounts. Von 
Neumann asked for the maximum rate ¢ at which the economy can expand and still stay 
nonnegative, meaning that u; > tug > 0. 

Thus the problem requires u; > tAu,. It is like the Perron-Frobenius theorem, with A 
on the other side. As before, equality holds when t reaches tmax—which is the eigenvalue 


associated with the positive eigenvector of A~!. In this example the expansion factor is 
10. 


1 4 0 1] {1 0.9 9 
x= 15 and Ax 0). ot) 384. 6S = (454 = To* 
5 ek LI. 45 


With steel-food-labor in the ratio 1-5-5, the economy grows as quickly as possible: The 
maximum growth rate is 1/A,. 
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Problem Set 5.3 


. Prove that every third Fibonacci number in 0,1,1.2,3,... is even. 


. Bernadelli studied a beetle “which lives three years only. and propagates in as third 
year.” They survive the first year with probability 7 and the second with probability 
i and then produce six females on the way out: 


0 0 6 
Beetle matrix A=/5 0 0 
0 30 


Show that A? = J, and follow the distribution of 3000 beetles for six years. 


. For the Fibonacci matrix A = [| |] , compute A’, A®, and A*. Then use the text and 
a calculator to find Fy. 


. Suppose each “Gibonacci” number G; is the average of the two previous numbers 
Gy, and Gx. Then Gy42 = 5 (Gri + Gx): 


Guyo = 5 G41 + 5Gp 
Gey = Gest 


Gey 
Gx 


Gr+2 


-( 


Gr 


(a) Find the eigenvalues and eigenvectors of A. 
(b) Find the limit as n — ce of the matrices A” = SA"S~!. 


(c) If Gg = 0 and G; = 1, show that the Gibonacci numbers approach é. 


. Diagonalize the Fibonacci matrix by completing S~!: 


EE U1 


Do the multiplication SA‘ S~! al to find its second component. This is the Ath Fi- 
bonacci number Fy = (AMAS) /(AytA2). 


. The numbers Ae and Ne satisfy the Fibonacci rule Fy. = Fy) + Fr: 
ABZ aR yak and | ARP SAR Lae, 


Prove this by using the original equation for the A’s (multiply it by A). Then any 
combination of Af and AS satisfies the rule. The combination hj = (Af — A¥) /(Ay — 
Az) gives the right start of Fo = 0 and F, = 1. 


10. 


11. 


12. 


13. 
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Lucas started with Lo = 2 and L; = 1. The rule Ly42 = Lyi; + Lz is the same, so A 
is still Fibonacci’s matrix. Add its eigenvectors x; + x2: 
le |e 


AEBS 


1 1 2 
Multiplying by A‘, the second component is Ly = AX +A‘. Compute the Lucas 
number Lj9 slowly by Lyi2 = Ly41 + Ly, and compute approximately by Aas 


Suppose there is an epidemic in which every month half of those who are well be- 
come sick, and a quarter of those who are sick become dead. Find the steady state 
for the corresponding Markov process 


dks 1 4 0| |& 
= 3 1 

Sk4] | = 0 4 3 Sk 
1 

Wk+1 O O 5] | we 


Write the 3 by 3 transition matrix for a chemistry course that is taught in two sections, 
if every week t of those in Section A and ; of those in Section B drop the course, 
and é of each section transfer to the other section. 


Find the limiting values of y, and ; (k — ©) if 


Yer = By +.3z% youd 
Zery = .2yp + .72% 2 =: 


Also find formulas for y; and z, from Ak = SAKS~!. 


(a) From the fact that column 1 + column 2 = 2(column 3), so the columns are 
linearly dependent find one eigenvalue and one eigenvector of A: 


Ae a3 
A=|4 2 3 
4 4 4 


(b) Find the other eigenvalues of A (it is Markov). 

(c) If uy = (0, 10,0), find the limit of Akug as k — ©. 

Suppose there are three major centers for Move-It- Yourself trucks. Every month half 
of those in Boston and in Los Angeles go to Chicago, the other half stay here they 
are, and the trucks in Chicago are split equally between Boston and Los Angeles Set 


up the 3 by 3 transition matrix A, and find the steady state u.. corresponding to the 
eigenvalue A = 1. 


(a) In what range of a and b is the following equation a Markov process? 


a b 1 
Us| = Aug = eer ee ba: lead 
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14. 


15. 


16. 


17. 
18. 


19. 
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(b) Compute uz = SAKS—! uo for any a and b. 


(c) Under what condition on a and b does uz approach a finite limit as k — o, and 
what is the limit? Does A have to be a Markov matrix? 


Multinational companies in the Americas, Asia, and Europe have assets of $4 trillion. 
At the start, $2 trillion are in the Americas and $2 trillion in Europe. Each year 4 the 
American money stays home, and i goes to each of Asia and Europe. For Asia and 
Europe, 5 stays home and 5 is sent to the Americas. 


(a) Find the matrix that gives 


Americas Americas 
Asia =A Asia 
Europe dl Europe ne 


(b) Find the eigenvalues and eigenvectors of A. 

(c) Find the limiting distribution of the $4 trillion as the world ends. 

(d) Find the distribution of the $4 trillion at year k. 

If A is a Markov matrix, show that the sum of the components of Ax equals the sum 


of the components of x. Deduce that if Ax = Ax with A # 1, the components of the 
eigenvector add to zero. 


The solution to du/dt = Au = ie | u (eigenvalues i and —i) goes around in a circle: 
u = (cost, sint). Suppose we approximate du/dt by forward, backward, and centered 
differences F, B, C: 


(F) uni — Un = Aly OF Un+1 = (I +A)uy (this is Euler’s method). 
(B) Uns] — Un = Aun 41 OF Un, = (I—A)~ un (backward Euler). 
(C) Un41 — Un = 5A(Uns1 + un) OF Ung = (FE — 5A) -1(E+ 5A)Un. 


Find the eigenvalues of J+A, (I1A)~!, and (I— 4A)~!(1+ 5A). For which difference 
equation does the solution u,, stay on a circle? 


What values of @ produce instability in vy +1 = (Vn +Wn), Wn41 = A(Vn + Wn)? 


Find the largest a, b, c for which these matrices are stable or neutrally stable: 


Pos ae eet 


Multiplying term by term, check that (J1A)(I +A+A*+---) =I. This series rep- 
resents (/1A)~!. It is nonnegative when A is nonnegative, provided it has a finite 
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sum; the condition for that is Amax < 1. Add up the infinite series, and confirm that 


it equals ([1A)~', for the consumption matrix 


1 1 
0 1 which has Amax = 0. 
0 0 


20. For A = [} 2], find the powers A‘ (including A°) and show explicitly that their sum 
agrees with (J—A)~!. 


21. Explain by mathematics or economics why increasing the “consumption matrix” A 
must increase tmax = A; (and slow down the expansion). 


22. What are the limits as k — o (the steady states) of the following? 


ik 4.2.6 |i) 4.2.6 3}. 


Problems 23-29 are about A = SAS~! and A‘ = SA‘S~! 


4.2.6 3] 


23. Diagonalize A and compute SA‘S~! to prove this formula for A*: 


oe has gee? 
2 3 2 


5kiq 5k] 


A= 
5k 1 5k 4] 


24. Diagonalize B and compute SA‘S~! to prove this formula for B*: 


3. 1 3k 3k 9k 
B= ee i 

0 2 0 2 
25. The eigenvalues of A are | and 9, the eigenvalues of B are t1 and 9: 
5 4 4 

and B= 2 

4 5 5 4 
Find a matrix square root of A from R = S\/AS~!, Why is there no real matrix square 
root of B? 


has B= | 


A= 


26. If A and B have the same A’s with the same full set of independent eigenvectors, their 
factorizations into are the same. SoA = B. 


27. Suppose A and B have the same full set of eigenvectors, so that A= SA,S~! and 
B =SA>S~!. Prove that AB = BA. 


28. (a) When do the eigenvectors for A = 0 span the nullspace N(A)? 
(b) When do all the eigenvectors for A 4 0 span the column space C(A)? 
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29. The powers A‘ approach zero if all |A;| < 1, and they blow up if any |A,| > 1. Peter 
Lax gives four striking examples in his book Linear Algebra. 
5 6.9 
—3 -—4 


7 
Bo a2 B= os 2 Ge 3) ae 
1 4 —5 -3 —3 —4 

}A1074 || = 10/7 B i024 —] (024 en 6 \)p1074 || e 10~78 
Find the eigenvalues A = e’® of B and C to show that B+ = J and C’ = —I. 


AS 


5.4 Differential Equations and e“’ 


Wherever you find a system of equations, rather than a single equation, matrix theory 
has a part to play. For difference equations, the solution uz = Aug depended on the owen 
of A. For differential equations, the solution u(t) = e“‘u(0) depends on the exponential 
of A. To define this exponential. and to understand it, we turn right away to an example: 


are | U. (1) 


Differential equation ag = Au = 
dt 1 -—2 


The first step is always to find the eigenvalues (11 and —3) and the eigenvectors: 


=(-l) Hl and A = (—3) fal 


1 
Then several approaches lead to u(t). Probably the best is to match the general solution 
to the initial vector u(0) at t = 0. 

The general solution is a combination of pure exponential solutions. These are so- 
lutions of the special form ce*'x, where A is an eigenvalue of A and x is its eigenvec- 
tor. These pure solutions satisfy the differential equation, since d/dt(ce*!x) = A(ce*'x). 
(They were our introduction to eigenvalues at the start of the chapter.) In this 2 by 2 
example, there are two pure exponentials to be combined: 


1 
1 


1 
—1 


A 


1 1 x 
Solution u(t) = ce'x, teoex. or ux E i f 7 "| . (2) 
— e 
At time zero, when the exponentials are e® = 1, u(0) determines c, and co: 
C2 


1 1 
Initial condition u(O) = cyx, + Cox2 = é i "| = Sc. 


You recognize S, the matrix of eigenvectors. The constants c = S~'u(0) are the same as 
they were for difference equations. Substituting them back into equation (2), the solution 
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o-f JF olflok oem © 


Here is the fundamental formula of this section: Se“S~'u(0) solves the differential 
equation, just as SA‘S~!ug solved the difference equation: 


=] e! 
‘| and eV = | | 2 eS 


There are two more things to be done with this example. One is to complete the 
mathematics, by giving a direct definition of the exponential of a matrix. The other 
is to give a physical interpretation of the equation and its solution. It is the kind of 
differential equation that has useful applications. 


u(t) =SeS~'u(0) with A= 


The exponential of a diagonal matrix A is easy; e“’ just has the n numbers e*’ on 


the diagonal. For a general matrix A, the natural idea is to imitate the power series 
et = 14x4x7/2!42°/3!+4---. If we replace x by At and | by J, this sum is ann by n 


matrix: 


(Ar)? (At)? 
oa! 


The series always converges, and its sum e“” has the right properties: 


Matrix exponential et —T+At+ 


(5) 


d 
(e45)(A) = (ACH), (e")(e4") = 1, and qe) = Ae”. (6) 
From the last one, u(t) = e““u(0) solves the differential equation. This solution must 
be the same as the form Se“’S~!u(0) used for computation. To prove directly that those 
solutions agree, remember that each power (SAS~!)* telescopes into A‘ = SA‘S~! (be- 
cause S~! cancels S). The whole exponential is diagonalized by S: 


SA2S—'t2, SA3S—143 
2! = 3! 
(Ar)? (Ar)? 


= . \ el — @ Aterl 
=s(1+ar+ 7 = 31 ae )s =Se"S ~. 


ef = 71+ SAS" !t+ 


Example 1. In equation (1), the exponential of A= |=? ',] has A= |! _,]: 


—t -_ 
hi etme 2 Ee ‘ 1 1 
is ene) lak oa 


Att =0 we get e? = J. The infinite series e“’ gives the answer for all t, but a series can be 
hard to compute. The form Se“’S~! gives the same answer when A can be diagonalized; 
it requires n independent eigenvectors in S. This simpler form leads to a combination of 
n exponentials e*'x—which is the best solution of all: 
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5L If A can be diagonalized, A = SAS~!, then du/dt = Au has the solution 


u(t) = e““u(0) = Se’S~!u(0). (7) 
The columns of S are the eigenvectors x;,...,x, of A. Multiplying gives 
ert 
u(t) = [x1 +++ Xn te s~'u(0) 
It (8) 
e n 


= cye"'x, ++» +c,e"'x, = combination of ex. 


The constants c; that match the initial conditions u(0) are c = S~'u(0). 

This gives a complete analogy with difference equations and SAS~!ug. In both cases 
we assumed that A could be diagonalized. since otherwise it has fewer than n eigenvec- 
tors and we have not found enough special solutions. The missing Solutions do exist, 
but they are more complicated than pure exponentials e*’x. They involve “generalized 
eigenvectors” and factors like te*". (To compute this defective case we can use the Jor- 
dan form in Appendix B, and find e”’.) The formula u(t) = e“’u(0) remains completely 
correct. 

The matrix e“’ is never singular. One proof is to look at its eigenvalues; if A is an 
eigenvalue of A, then e”” is the corresponding eigenvalue of e4’—and e”’ can never be 
zero. Another approach is to compute the determinant of the exponential: 


dete”! = elit phat . -ehnt — eltace(Ar) | (9) 


At as its inverse. 


Quick proof that e“’ is invertible: Just recognize e~ 
This invertibility is fundamental for differential equations. If n solutions are linearly 
independent at t = 0, they remain linearly independent forever. If the initial vectors are 


V1,---;¥n, We can put the solutions e“'y into a matrix: 


The determinant of the left-hand side is the Wronskian. It never becomes zero, because 
it is the product of two nonzero determinants. Both matrices on the right-hand side are 
invertible. 


Remark. Not all differential equations come to us as a first-order system du/dt = Au. 
We may start from a single equation of higher order, like y’” — 3y’ + 2y’ = 0. To convert 
to a 3 by 3 system, introduce v = y’ and w = v’ as additional unknowns along with y 
itself. Then these two equations combine with the original one to give u’ = Au: 


y =v 0 1 O; ly 
v=w or u=|0 0 1| |v] =Aw. 
w’ = 3w—2v OS. 3h ap 
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Figure 5.1: A model of diffusion between four segments. 


We are back to a first-order system. The problem can be solved two ways. In a course 
on differential equations, you would substitute y = e*” into y” — 3y” + 2y’ =0: 


(A2—3A7+2A)e#=0 or A(A—1)(A—2)e* =0. (10) 


The three pure exponential solutions are y = e’, y = e’, and y = e”’. No eigenvectors are 
involved. In a linear algebra course, we find the eigenvalues of A: 


—A~A 1 0 
det(A-AN=]0 -A TF | =-A?+34?-21=0. (11) 
Oy 22: Bx 


Equations (10) and (11) are the same! The same three exponents appear: A = 0, A = 1, 
and A = 2. This is a general rule which makes the two methods consistent; the growth 
rates of the solutions stay fixed when the equations change form. It seems to us that 
solving the third-order equation is quicker. 


The physical significance of du/dt = len a | u is easy to explain and at the same 
time genuinely important. This differential equation describes a process of diffusion. 
Divide an infinite pipe into four segments (Figure 5.1). At time t = 0, the middle seg- 
ments contain concentrations v(0) and w(0) of a chemical. At each time t, the diffusion 
rate between two adjacent segments is the difference in concentrations. Within each 
segment, the concentration remains uniform (zero in the infinite segments). The process 
is continuous in time but discrete in space; the unknowns are v(t) and w(t) in the two 
inner segments S; and S». 

The concentration v(t) in S; is changing in two ways. There is diffusion into So, and 
into or out of Sz. The net rate of change is dv/dt, and dw/dt is similar: 


d 
Flow rate into S| A = (w—v)+(0-v) 


d 
Flow rate into S» — = (0—w)+(v—-w). 

This law of diffusion exactly matches our example du/dt = Au: 

v du —2 | 

u= and —'= U. 

w dt 1 -2 
The eigenvalues —1 and —3 will govern the solution. They give the rate at which the 
concentrations decay, and A; is the more important because only an exceptional set of 


—2v+w 
vy—2w 
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starting conditions can lead to “superdecay” at the rate e~*’, In fact, those conditions 
must come from the eigenvector (1,—1). If the experiment admits only nonnegative 
concentrations, superdecay is impossible and the limiting rate must be e~’. The solution 
that decays at this slower rate corresponds to the eigenvector (1,1). Therefore the two 
concentrations will become nearly equal (typical for diffusion) as t — 9. 

One more comment on this example: It is a discrete approximation, with only two 
unknowns, to the continuous diffusion described by this partial differential equation: 
du O-u 
Ot axe 
That heat equation is approached by dividing the pipe into smaller and smaller segments, 
of length 1/N. The discrete system with N unknowns is governed by 


Heat equation 


=e | = | | | = Au. (12) 


uN 1 -—2 un 


This is the finite difference matrix with the 1, —2, | pattern. The right side Au approaches 
the second derivative d7u/dx’, after a scaling factor N7 comes from the flow problem. 
In the limit as N — , we reach the heat equation du/dt = 0*u/dx?. Its solutions 
are still combinations of pure exponentials, but now there are infinitely many. Instead 
of eigenvectors from Ax = Ax, we have eigenfunctions from d*u/dx* = Au. Those are 
u(x) = sinnzx with A = —n?x?. Then the solution to the heat equation is 


u(t)= >) cre” *' sinnmx. 
n=1 


The constants c, are determined by the initial condition. The novelty is that the eigen- 
vectors are functions u(x), because the problem is continuous and not discrete. 


stability of differential equations 


Just as for difference equations. the eigenvalues decide how u(t) behaves as t — 9. 
As long as A can be diagonalized, there will be n pure exponential solutions to the 
differential equation, and any specific solution u(t) is some combination 


u(t) = Sets ug = cyehtxy +e + cgeklttxy. 


Stability is governed by those factors e®"". If they all approach zero, then u(t) approaches 
zero: if they all stay bounded, then u(r) stays bounded; if one of them blows up, then 
except for very special starting conditions the solution will blow up. Furthermore, the 
size of e*’ depends only on the real part of A. It is only the real parts of the eigenvalues 
that govern stability: If A = a+ ib, then 

eM 


= e“ elt — 6 (cosbt +isinbt) and the magnitude is |e*’| =e. 
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This decays for a < 0, it is constant for a = 0, and it explodes for a > 0. The imaginary 
part is producing oscillations, but the amplitude comes from the real part. 

5M The differential equation du/dt = Au is 

stable and e“' —> 0 whenever all ReA; < 0, 

neutrally stable when all Red; < 0 and Red; = 0, and 

unstable and e“ is unbounded if any eigenvalue has Red; > 0. 
In some texts the condition Red < 0 is called asymptotic stability, because it guarantees 
decay for large times ¢. Our argument depended on having n pure exponential solutions, 
but even if A is not diagonalizable (and there are terms like te’) the result is still true: 


All solutions approach zero if and only if all eigenvalues have Red < 0. 
Stability is especially easy to decide for a 2 by 2 system (which is very common in 


applications). The equation is 
du _ ja b ; 
dt lc d 


and we need to know when both eigenvalues of that matrix have negative real parts. 
(Note again that the eigenvalues can be complex numbers.) The stability tests are 


Red; <0 The trace a+d must be negative. 
Redz < 0 The determinant ad — bc must be positive. 


When the eigenvalues are real, those tests guarantee them to be negative. Their product 
is the determinant; it is positive when the eigenvalues have the same sign. Their sum is 
the trace; it is negative when both eigenvalues are negative. 

When the eigenvalues are a complex pair x + iy, the tests still succeed. The trace 
is their sum 2x (which is < 0) and the determinant is (x + iy)(x —iy) =x? +y? > 0. 
Figure 5.2 shows the one stable quadrant, trace < 0 and determinant > 0. It also shows 
the parabolic boundary line between real and complex eigenvalues. The reason for the 
parabola is in the quadratic equation for the eigenvalues: 


det |“ M2 — A? — (trace)A + (det) = 0. (13) 
Ci ih 


The quadratic formula for A leads to the parabola (trace)* = 4(det): 


A, and Az = ; trace Se V (trace)? — s(det) (14) 


Above the parabola, the number under the square root is negative—so A is not real. On 
the parabola, the square root is zero and A is repeated. Below the parabola the square 
roots are real. Every symmetric matrix has real eigenvalues, since if b = c, then 


(trace)* — 4(det) = (a+d)* — 4(ad —b*) = (a—d)? + 4b > 0. 
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Adeterminant D 


M = ro and 
‘ both Rei < 0 both ReA > 0 T2 AD 
<a stable unstable es 
sa complex eigenvalues oe 

bothA<0 0. vf poth A> 0 
real and stable fi : es . real and unstable 
aoe Snail > 
trace T’ 


det < 0 gives A; < 0 and Az > 0: real and unstable 


Figure 5.2: Stability and instability regions for a 2 by 2 matrix. 


For complex eigenvalues, b and c have opposite signs and are sufficiently large. 


Example 2. One from each quadrant: only #2 is stable: 


pa le A fal [oy 


On the boundaries of the second quadrant, the equation is neutrally stable. On the hori- 
zontal axis, one eigenvalue is zero (because the determinant is A;A2 = 0). On the vertical 
axis above the origin, both eigenvalues are purely imaginary (because the trace is Zero). 
Crossing those axes are the two ways that stability is lost. 


The n by n case is more difficult. A test for ReA; < 0 came from Routh and Hurwitz, 
who found a series of inequalities on the entries a;;. I do not think this approach is 
much good for a large matrix; the computer can probably find the eigenvalues with more 
certainty than it can test these inequalities. Lyapunov’s idea was to find a weighting 
matrix W so that the weighted length \|Wu(t)|| is always decreasing. If there exists 
such a W, then ||Wu|| will decrease steadily to zero, and after a few ups and downs u 
must get there too (stability). The real value of Lyapunov’s method is for a nonlinear 
equation—then stability can be proved without knowing a formula for u(t). 


Example 3. du/dt = [° ||| u sends u(t) around a circle, starting from w(0) = (1,0). 
Since trace = 0 and det = 1, we have purely imaginary eigenvalues: 


i Gy] eaet=o SO A =-+iand —i. 


The eigenvectors are (1,—i) and (1,7). and the solution is 


1 
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That is correct but not beautiful. By substituting cost +isint for e” and e~", real num- 
bers will reappear: The circling solution is u(t) = (cost, sinf). 
Starting from a different u(0) = (a,b), the solution u(t) ends up as 


acost — bsint cost —sint| |a 
u(t) = Pe | ee : (15) 

bcost +asint sint cost | |b 
There we have something important! The last matrix is multiplying u(0), so it must be 
the exponential e4’. (Remember that u(t) = e““u(0).) That matrix of cosines and sines 


is our leading example of an orthogonal matrix. The columns have length 1, their inner 
product is zero, and we have a confirmation of a wonderful fact: 


If A is skew-symmetric (A! = —A) then e“ is an orthogonal matrix. 


A! = —A gives a conservative system. No energy is lost in damping or diffusion: 
AT=-A,  (e)P =e, and |le““u(0)|| = ||u(0)|- 


That last equation expresses an essential property of orthogonal matrices. When they 
multiply a vector, the length is not changed. The vector u(0) is just rotated, and that 
describes the solution to du/dt = Au: It goes around in a circle. 

In this very unusual case, e“’ can also be recognized directly from the infinite series. 
Note that A = ie al has A* = —I, and use this in the series for e“’: 


2 3 
At)2 At)? ( -F 4+.) (-1+5----] 
ey (ia ad are aad : 7 
2 6 (*-F +--+) ( -F+--) 
__ |cost —sint 
~ |sint cost 


—2 
Example 4. The diffusion equation is stable: A = 


a has A = —landdA = -3. 


Example 5. If we close off the infinite segments, nothing can escape: 


du_|{-1 1 dv/dt =w—v 
dt |1 —-1 dw/dt =v—w. 


This is a continuous Markov process. Instead of moving every year, the particles move 
every instant. Their total number v+ w is constant. That comes from adding the two 
equations on the right-hand side: the derivative of v + w is zero. 

A discrete Markov matrix has its column sums equal to Amax = 1. A continuous 
Markov matrix, for differential equations, has its column sums equal to Amax = 0. A is 
a discrete Markov matrix if and only if B = A —/ is a continuous Markov matrix. The 
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Figure 5.3: The slow and fast modes of oscillation. 


steady state for both is the eigenvector for Amax. It is multiplied by 1‘ = 1 in difference 
equations and by e% = 1 in differential equations, and it doesn’t move. 
In the example, the steady state has v = w. 


Example 6. In nuclear engineering, a reactor is called critical when it is neutrally 
stable; the fission balances the decay. Slower fission makes it stable, or subcritical, and 
eventually it runs down. Unstable fission is a bomb. 


Second-Order Equations 


The laws of diffusion led to a first-order system du/dt = Au. So do a lot of other appli- 
cations, in chemistry, in biology, and elsewhere, but the most important law of physics 
does not. It is Newton’s law F = ma, and the acceleration a is a second derivative. In- 
ertial terms produce second-order equations (we have to solve d*u/dt? = Au instead of 
du/dt = Au), and the goal is to understand how this switch to second derivatives alters 
the solution>. It is optional in linear algebra, but not in physics. 

The comparison will be perfect if we keep the same A: 


2 - 
au 2 AC re 


woe 


| 


Two initial conditions get the system started—the “displacement” u(0) and the “veloc- 
ity” u’(0). To match these conditions, there will be 2n pure exponential solutions. 

Suppose we use @ rather than A, and write these special solutions as u = e'®’x. Sub- 
stituting this exponential into the differential equation, it must satisfy 

d? iot i@t 2 

7rAG x) =A(e"™x), or — @°x = Ax. (17) 
The vector x must be an eigenvector of A, exactly as before. The corresponding eigen- 
value is now —@*, so the frequency @ is connected to the decay rate A by the law 


>Fourth derivatives are also possible, in the bending of beams, but nature seems to resist going higher than four. 
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—@* =A. Every special solution e*'x of the first-order equation leads to two special 
solutions e!@'x of the second-order equation. and the two exponents are @ = +/—A. 
This breaks down only when A = 0, which has just one square root; if the eigenvector is 
x, the two special solutions are x and tx. 

For a genuine diffusion matrix, the eigenvalues A are all negative and the frequencies 
it produce 
neutral stability, the solution neither grows or decays, and the total energy stays precisely 
constant. It just keeps passing around the system. The general solution to d*u/dt* = Au, 
if A has negative eigenvalues A),...,A, and if @j = JA; is 


@ are all real: Pure diffusion is converted into pure oscillation. The factors e 


u(t) = (cye'" +. dye) x1 + cdots + (creo + dye") Xn. (18) 


As always, the constants are found from the initial conditions. This is easier to do (at the 
expense of one extra formula) by switching from oscillating exponentials to the more 
familiar sine and cosine: 


u(t) = (a, cos @jt + by sin @it)x1 +--+ + (dy COS Wyt + by Sin Wyt) Xp. (19) 


The initial displacement u(0) is easy to keep separate: t = 0 means that sin @t = 0 and 
cos ot = 1, leaving only 


u(O) =aixy+-*++anxXn, or u(0)=Sa, or a=S~'u(0). 


Then differentiating u(r) and setting t = 0. the b’s are determined by the initial velocity: 
u' (0) = b,x, +++: +b_,@pxp. Substituting the a’s and b’s into the formula for u(t), the 
equation is solved. 

The matrix A = [ae i has A; = —1 and A = —3. The frequencies are @, = 1 and 
@ = V3. If the system starts from rest, u’(0) = 0, the terms in bsin wt will disappear: 


1 1 
Solution from u(0) = | u(t) = 5 cost 


1 1 

1 =I 5008 V3t | 

Physically, two masses are connected to each other and to stationary wails by three 
identical springs (Figure 5.3). The first mass is held at v(O) = 1, the second mass is held 
at w(0) = 0, and at t = 0 we let go. Their motion u(t) becomes an average of two pure 
oscillations, corresponding to the two eigenvectors. In the first mode x; = (1,1), the 
masses move together and the spring in the middle is never stretched (Figure 5.3a). The 
frequency @, = | is the same as for a single spring and a single mass. In the faster mode 
x2 = (1,—1) with frequency \/3, the masses move oppositely but with equal speeds. The 
general solution is a combination of these two normal modes. Our particular solution is 
half of each. 

As time goes on, the motion is “almost periodic.” If the ratio @;/@ had been a 
fraction like 2/3, the masses would eventually return to u(0) = (1,0) and begin again. 
A combination of sin2r and sin3t would have a period of 27. But \/3 is irrational. The 
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best we can say is that the masses will come arbitrarily close to (1,0) and also (0,1). 
Like a billiard ball bouncing forever on a perfectly smooth table, the total energy is fixed. 
Sooner or later the masses come near any state with this energy. 

Again we cannot leave the problem without drawing a parallel to the continuous case. 
As the discrete masses and springs merge into a solid rod, the “second differences” given 
by the 1, —2, 1 matrix A turn into second derivatives. This limit is described by the 
celebrated wave equation 07u/dt? = 0*u/dx?. 


Problem Set 5.4 


1. Following the first example in this section, find the eigenvalues and eigenvectors, 
and the exponential e“’, for 
—-1 1 
1 -1|- 


2. For the previous matrix, write the general solution to du/dt = Au, and the specific 
solution that matches u(0) = (3,1). What is the steady state as t — 0? (This is a 
continuous Markov process; A = 0 in a differential equation corresponds to A = 1 in 
a difference equation, since e = 1.) 


3. Suppose the time direction is reversed to give the matrix —A: 


du i ‘| i 
= u with uy = Fils 


dt |-1 1 
Find u(t) and show that it blows up instead of decaying as t — o. (Diffusion is 
irreversible, and the heat equation cannot run backward.) 


4. If Pis a projection matrix, show from the infinite series that 
e” ~I+1.718P. 


t+T) _ pAt 


5. A diagonal matrix like A = [} 9] satisfies the usual rule e* e\’ because 


the rule holds for each diagonal entry. 


(a) Explain why e4¢+7) = e4e4T, using the formula e“’ = Se4’s—!, 
(b) Show that e4+8 = e4e® is not true for matrices, from the example 


gez|2 9 
1 0 


0 -1l 
B= ; 0 | (use series for e and &? ). 


10. 


11. 
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The higher order equation y” + y = 0 can be written as a first-order system by intro- 
ducing the velocity y’ as another unknown: 


le 


If this is du/dt = Au, what is the 2 by 2 matrix A? Find its eigenvalues and eigen- 
vectors, and compute the solution that starts from y(0) = 2, y’(0) = 0. 


Convert y” = 0 to a first-order system du/dt = Au: 


sH-6-640) 


This 2 by 2 matrix A has only one eigenvector and cannot be diagonalized. Compute 
e“ from the series J++ At +--+ and write the solution e““u(0) starting from y(0) = 3, 
y'(O) = 4. Check that your (y, y’) satisfies y” = 0. 


. Suppose the rabbit population 7 and the wolf population w are governed by 


Fame 
dw = 
—=r+w 
dt 


(a) Is this system stable, neutrally stable, or unstable? 
(b) If initially r = 300 and w = 200, what are the populations at time ft? 


(c) After a long time, what is the proportion of rabbits to wolves? 


. Decide the stability of u’ = Au for the following matrices: 


23 1 2 
(a) A= 4 Ar (b) A= 3 al 
1 1 —1 -l 
(c) A= 1 | (d) A= 4 é | 


Decide on the stability or instability of dv/dt = w, dw/dt = v. Is there a solution 
that decays? 


From their trace and determinant, at what time ¢ do the following matrices change 
between stable with real eigenvalues, stable with complex eigenvalues, and unstable? 


{i-=4 0 41 | 
: } ah 2 ‘ ai a= i 
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12. 


13. 


14. 


15. 


16. 


17. 


18. 


19. 
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Find the eigenvalues and eigenvectors for 


0 3 =O 
du 
—=Au=|]-3 0 4I/4u 
dt 

0 -4 0 


Why do you know, without computing, that e4’ will be an orthogonal matrix and 
||e(t) ||? = uf + u3 + v3 will be constant? 


For the skew-symmetric equation 


b -—-a O U3 
(a) write out uw), v4, uv and confirm that ww; + u5u2 +u4u3 = 0. 
(b) deduce that the length uw} + u5 + v3 is a constant. 
(c) find the eigenvalues of A. 


The solution will rotate around the axis w = (a,b,c), because Au is the “cross prod- 
uct” u x w—which is perpendicular to u and w. 


What are the eigenvalues A and frequencies @, and the general solution, of the fol- 
lowing equation? 

du |-5 4 

de |4 —5|" 


Solve the second-order equation 


du |-5 -1 1 hee 10 
a= | a with «o)=|y| and «=| 


In most applications the second-order equation looks like Mu" + Ku = 0, with a mass 
matrix multiplying the second derivatives. Substitute the pure exponential u = e'®"x 
and find the “generalized eigenvalue problem” that must be solved for the frequency 
@ and the vector x. 


With a friction matrix F in the equation vu’ + Fu’ — Au = 0, substitute a pure expo- 


nential u = e”’x and find a quadratic eigenvalue problem for A. 


For equation (16) in the text, with @ = | and V3, find the motion if the first mass is 
hit at t = 0; u(0) = (0,0) and u’(0) = (1,0). 


Every 2 by 2 matrix with trace zero can be written as 


a b+e 
b=6. Sas" 


Show that its eigenvalues are real exactly when a* + b* > c’. 


20. 


21. 


22. 


23. 


24. 


22s 
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By back-substitution or by computing eigenvectors, solve 


d 121 1 

u 

aa 0 3 6|u with u(0O) = |0 
00 4 1 

Find A’s and x’s so that u = e*x solves 


du _ {4 3 . 
dt |0 1 
What combination u = cye"""x; + c2e*2'x> starts from u(0) = (5,—2)? 


Solve Problem 21 for u(t) = (y(t),z(t)) by back-substitution: 
dz 
First solve nn starting from = z(0) = —2. 


d 
Then solve = =4y+3z, startingfrom y(0)=5. 


The solution for y will be a combination of e™ and e’. 


Find A to change y” = 5y’ + 4y into a vector equation for u(t) = (y(t), y’(t)): 


=-El-( I 


What are the eigenvalues of A? Find them also by substituting y = e”” into the scalar 
equation y” = Sy’ +4y. 


A door is opened between rooms that hold v(0) = 30 people and w(0) = 10 people. 
The movement between rooms is proportional to the difference v — w: 


dv a4 dw 
—=w-v —_— = 
dt dt 

Show that the total v + w is constant (40 people). Find the matrix in du/dt = Au, and 

its eigenvalues and eigenvectors. 


What are v and w att = 1? 


Reverse the diffusion of people in Problem 24 to du/dt = —Au: 


dv pas dw 

—=Vv-w —=w-v. 

dt dt 
The total v+ w still remains constant. How are the 2’s changed now that A is changed 
to —A? But show that v(r) 


grows to infinity from v(0) = 30. 
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26. The solution to y” = 0 is a straight line y = C + Dr. Convert to a matrix equation: 


y}_ |O 1} Jy y} ar | ¥(0) 
; = f 4 A has the solution Hl = ¢4 ke (0) 


This matrix A cannot be diagonalized. Find A? and compute e“! = J + At + 5 At? ar 
--, Multiply your e” times (y(0),y/(0)) to check the straight line y(t) = y(0) + 
y'(O)t. 
27. Substitute y = e”” into y” = 6y' — 9y to show that A = 3 is a repeated root. This is 
trouble; we need a second solution after e*’. The matrix equation is 


ml" [> db} 


Show that this matrix has A = 3,3 and only one line of eigenvectors. Trouble here 
too. Show that the second solution is y = te*”. 


ad 
dt 


28. Figure out how to write my” + by’ + ky = 0 as a vector equation Mu’ = Au. 


29. (a) Find two familiar functions that solve the equation d*y/dt? = —y. Which one 
starts with y(0) = 1 and y’(0) = 0? 


(b) This second-order equation y’ = —y produces a vector equation u’ = Au: 
q p q 


Put y(t) from part (a) into u(t) = (y,y’). This solves Problem 6 again. 


= Au. 


30. A particular solution to du/dt = Au— bis up = A~'b, if A is invertible. The solutions 
to du/dt = Au give uy. Find the complete solution wu, + uy to 


+E 


31. If c is not an eigenvalue of A, substitute uv = e“v and find v to solve du/dt = Au — 
e'b. This u = e“y is a particular solution. How does it break down when c is an 
eigenvalue? 


du du 
— =2u—8. = 
(a) Ti u—8 (b) i 


32. Find a matrix A to illustrate each of the unstable regions in Figure 5.2: 


(a) Ay <Oand A, > 0. 
(b) Ay > 0 and Ax > 0. 
(c) Complex A’s with real part a > 0. 


Problems 33-41 are about the matrix exponential e“’. 


33. 


34. 


35. 


36. 
37. 


38. 


39. 


40. 


41. 


42. 


43. 
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Write five terms of the infinite series for e“’. Take the t derivative of each term. Show 
that you have four terms of Ae’. Conclusion: e“‘u(0) solves u! = Au. 


The matrix B = |? > | has B? = 0. Find e* from a (short) infinite series. Check that 


the derivative of e?" is Be®*. 


Starting from u(0), the solution at time T is e47 u(0). Go an additional time t to reach 
e“'(e4Tu(0)). This solution at time t + 7 can also be written as . Conclusion: 
e“' times e“7 equals 


Write A = E al in the form SAS~!. Find e“’ from Se“’S—!. 


If A? = A, show that the infinite series produces e’ = +(e! —1)A. For A= [4 4] in 
Problem 36, this gives e4” = 


Generally ee? is different from e?e4. They are both different from e4+?. Check this 
using Problems 36—37 and 34: 

1 0 

0 O|- 


1 1 — 
poe ae 
0 0 0 O 


Write A = [}4] as SAS~!. Multiply Se“S~' to find the matrix exponential e”’. 
Check e“’ = J when t = 0. 


Put A = |} 3] into the infinite series to find e“’. First compute A*: 


Av! Oe 3 de jecees es 
“TO: T 0 0 Oe | 


Give two reasons why the matrix exponential e“’ is never singular: 


ee 
2 


(a) Write its inverse. 
(b) Write its eigenvalues. If Ax = Ax then e4’x = Xi: 


Find a solution x(t), y(t) of the first system that gets large as t — 0. To avoid this 
instability a scientist thought of exchanging the two equations! 


d. = Ze ao 
x/dt Ox Ay Peer dy/dt 2x + 2y 
dy/dt = —2x 4+ 2y dx/dt = Ox — Ay. 


Now the matrix ie = is stable. It has A < 0. Comment on this craziness. 


From this general solution to du/dt = Au, find the matrix A: 


2 1 
+ ce”! Hl : 


u(t) = cye~ 


312 Chapter 5 Eigenvalues and Eigenvectors 
5.5 Complex Matrices 


It is no longer possible to work only with real vectors and real matrices In the first half of 
this book, when the basic problem was Ax — b, the solution was real when A and b were 
real. Complex numbers could have been permitted. but would have contributed nothing 
new. Now we cannot avoid them. A real matrix has real coefficients in det(A — AJ), but 
the eigenvalues (as in rotations) may be complex. 

We now introduce the space C” of vectors with n complex components. Addition and 
matrix multiplication follow the same rules as before. Length is computed differently. 
The old way, the vector in C* with components (1,i) would have zero length: 17+ i =0, 
not good. The correct length squared is 17 + |i|? = 2. 

This change to ||x||7 = |x,|* +--+ |x,|? forces a whole series of other changes. The 
inner product, the transpose, the definitions of symmetric and orthogonal matrices, all 
need to be modified for complex numbers. The new definitions coincide with the old 
when the vectors and matrices are real. We have listed these changes in a table at the 
end of the section. and we explain them as we go. 

That table virtually amounts to a dictionary for translating real into complex. We 
hope it will be useful to the reader. We particularly want to find out about symmetric 
matrices and Hermitian matrices: Where are their eigenvalues, and what is special 
about their eigenvectors? For practical purposes, those are the most important questions 
in the theory of eigenvalues. We call attention in advance to the answers: 


1. Every symmetric matrix (and Hermitian matrix) has real eigenvalues. 


2. Its eigenvectors can be chosen to be orthonormal. 


Strangely, to prove that the eigenvalues are real we begin with the opposite possibility— 
and that takes us to complex numbers, complex vectors, and complex matrices. 


Complex Numbers and Their Conjugates 


Probably the reader has already met complex numbers; a review is easy to give. The 
important ideas are the complex conjugate x and the absolute value |x|. Everyone knows 
that whatever i is, it satisfies the equation i* = —1. It is a pure imaginary number, and 
so are its multiples ib; b is real. The sum a+ ib is a complex number, and it is plotted in 
a natural way on the complex plane (Figure 5.4). 

The real numbers a and the imaginary numbers ib are special cases of complex num- 
bers; they lie on the axes. Two complex numbers are easy to add: 


Complex addition (a+ib)+(c+id) = (a+c)+i(b+d). 
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imaginary axis 


a+ib= re” 


real axis 


complex conjugate 
a—ib=atib=re—% 


Figure 5.4: The complex plane, with a+ ib = re’® and its conjugate a—ib = re~'®. 


Multiplying a+ ib times c + id uses the rule that i? = —1: 


Multiplication = (a+ib)(c +id) = ac+ibe+ iad +i°bd 
= (ac — bd) +i(bc+ad). 
The complex conjugate of a+ ib is the number a— ib. The sign of the imaginary 
part is reversed. It is the mirror image across the real axis; any real number is its own 


conjugate, since b = 0. The conjugate is denoted by a bar or a star: (a+ib)* =a+ib= 
a— ib. It has three important properties: 


1. The conjugate of a product equals the product of the conjugates: 


(a+ib)(c+id) = (ac —bd) — i(be +-.ad) = (a+ib)(c+id). (1) 


2. The conjugate of a sum equals the sum of the conjugates: 


(a+c) +i(b+d) = (a+c)—i(b +d) = (a+ ib) + (c $id). 
3. Multiplying any a+ ib by its conjugate a — ib produces a real number a? + b?: 
Absolute value (a+ib)(a—ib)=a° +b’ =r’. (2) 
This distance r is the absolute value |a-+ib| = Va? +b?. 


Finally, trigonometry connects the sides a and b to the hypotenuse r by a = rcos@ 
and b = rsin@. Combining these two equations moves us into polar coordinates: 


Polar form a+ib=r(cos@ +isin@) = re’®. (3) 
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The most important special case is when r = 1. Then a+ ib is e!® = cos@ +isin@. It 
falls on the unit circle in the complex plane. As @ varies from 0 to 27, this number e’? 


circles around zero at the constant radial distance |e’®| = Vv cos? @ +sin?@ = 1. 

Example 1. x = 3+ 4i times its conjugate x = 3 — 47 is the absolute value squared: 
aX = (3+4i)(3 —47) =25=|x|?7 so r= |x| =5. 

To divide by 3+ 47, multiply numerator and denominator by its conjugate 3 — 4i: 


24+i 2413-41 10-5: 


344; 34473 S44; ~~ 5 


In polar coordinates, multiplication and division are easy: 


re'® times Re'™ has absolute value rR and angle 0 + a. 


re!® divided by Re'™ has absolute value r/R and angle 6 — a. 


Lengths and Transposes in the Complex Case 


We return to linear algebra, and make the conversion from real to complex. By definition, 
the complex vector space C" contains all vectors x with n complex components: 


x] 

x2 F ; 
Complex vector x= |, with components x; =a; + ibj;. 

Xn 


Vectors x and y are still added component by component. Scalar multiplication cx is 
now done with complex numbers c. The vectors vj,...,v, are linearly dependent if 
some nontrivial combination gives cjvj +...-+cgvz = 0; the c; may now be complex. 
The unit coordinate vectors are still in C”; they are still independent; and they still form 
a basis. Therefore C” is a complex vector space of dimension n. 

In the new definition of length, each xy is replaced by its modulus |x ;|*: 


Length squared Ilac||* = |au|? +--+ [xnl?. (4) 
1 2+i 

Example 2.x=| | and |ix||?=2;  y=|5 7). and [yl|? =25. 
l — 41 


For real vectors there was a close connection between the length and the inner product: 
\|x||? =x'x. This connection we want to preserve. The inner product must be modified 
to match the new definition of length, and we conjugate the first vector in the inner 
product. Replacing x by X, the inner product becomes 


Inner product xy = X1yi +--+ +XpYn. (5) 
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If we take the inner product of x = (1 +3i,3i) with itself, we are back to ||x||?: 
Length squared = Xx = (1+ )(1+4 4+ (31)(31) =2+9 and |[x||? = 11. 
Note that y'x is different from x! y; we have to watch the order of the vectors. 


This leaves only one more change in notation, condensing two symbols into one. 
Instead of a bar for the conjugate and a T for the transpose, those are combined into the 
conjugate transpose. For vectors and matrices, a superscript H (or a star) combines both 
operations. This matrix A’ =A = A* is called “A Hermitian”: 


“A Hermitian” A4=A' has entries (AM);; = Aji. (6) 


You have to listen closely to distinguish that name from the phrase “A is Hermitian,” 
which means that A equals A". If A is an m by n matrix, then AM is n by m: 


H 
2+i 3i 
Conjugate a : ke 4+i 0 
—]| —* : 
t —3i 5 O 
ranspose 6 i 


This symbol A®™ gives official recognition to the fact that, with complex entries, it is 
very seldom that we want only the transpose of A. It is the conjugate transpose A that 
becomes appropriate, and x! is the row vector [X; -+- Xy]. 


5N 


1. The inner product of x and y is xy. Orthogonal vectors have x4y = 0. 
2. The squared length of x is ||x||? = x4 x = |x|? +---+ |xp|?. 
3. Conjugating (AB)' = B™AT produces (AB)# = BUA, 


Hermitian Matrices 


We spoke in earlier chapters about symmetric matrices: A = Al. With complex entries, 
this idea of symmetry has to be extended. The right generalization is not to matrices that 
equal their transpose, but to matrices that equal their conjugate transpose. These are 
the Hermitian matrices, and a typical example is A: 


ze. > ee oe 


— A of 
3431 5 wD) 


Hermitian matrix A= 


The diagonal entries must be real; they are unchanged by conjugation. Each off-diagonal 
entry is matched with its mirror image across the main diagonal, and 3 — 3i is the conju- 
gate of 3 + 37. In every case, ajj = Giji. 

Our main goal is to establish three basic properties of Hermitian matrices. These 
properties apply equally well to symmetric matrices. A real symmetric matrix is cer- 
tainly Hermitian. (For real matrices there is no difference between A! and A4.) The 
eigenvalues of A are real—as we now prove. 
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Property 1 If A= A®, then for all complex vectors x, the number x#Ax is real. 


Every entry of A contributes to x4Ax. Try the 2 by 2 case with x = (u,v): 
z 3=31 
xHAx = E 7| | 


u 
34+3i 5 : 
= 2uu + Svv+ (3 —3i)mv+ (3+ 3i)uv 
= real + real + (sum of complex conjugates). 


For a proof in general. (x#Ax)# is the conjugate of the 1 by 1 matrix x#Ax, but we 
actually get the same number back again: (x4Ax)4 = xHAHyHH — HA. So that number 
must be real. 


Property 2 If A= A", every eigenvalue is real. 


Proof. Suppose Ax = Ax. The trick is to multiply by x": x#Ax = Axx. The left-hand 
side is real by Property 1, and the right-hand side xx = ||x||? is real and positive, because 
x #0. Therefore A = x4Ax/x4x must be real. Our example has A = 8 and A = —1: 


Rh: 33) 
343i 5S—-A 
=A?-7A-8 =(A-8)(A41). 


|A—Al| = =1?—7A4+10-|3—3i/? 


(8) 


Note. This proof of real eigenvalues looks correct for any real matrix: 


x!Ax 


False proof  Ax=Ax gives x'Ax=Ax'x, so A= —z— is real. 
LON 


There must be a catch: The eigenvector x might be complex. It is when A = A! that 
we can be sure A and x stay real. More than that, the eigenvectors are perpendicular: 
x'y = 0 in the real symmetric case and x4y = 0 in the complex Hermitian case. 


Property 3 Two eigenvectors of a real symmetric matrix or a Hermitian ma- 
trix, if they come from different eigenvalues, are orthogonal to one another. 


The proof starts with Ax = Ayx, Ay = Ayy, and A = AF: 
(Aix) My = (Ax) Fy = xFAy = x4 (apy). (9) 


The outside numbers are Ayx4y = Apx"y, since the ’s are real. Now we use the assump- 
tion Ay 4 As, which forces the conclusion that x"y = 0. In our example, 


(A—8I)x = —6 3-i] |x = 0) ge 1 
3+3i -—3 | |x 0}? 1+i 
3 3-3] ly] fo iy 
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These two eigenvectors are orthogonal: 


ea 
xy = E 1-i] | | ~(, 
—] 
Of course any multiples x/a@ and y/B are equally good as eigenvectors. MATLAB 
picks @ = ||x|| and B = ||y||, so that x/@ and y/B are unit vectors; the eigenvectors 
are normalized to have length 1. They are now orthonormal. If these eigenvectors are 


chosen to be the columns of S, then we have S~'AS = A as always. The diagonalizing 
matrix can be chosen with orthonormal columns when A = A®. 


In case A is real and symmetric, its eigenvalues are real by Property 2. Its unit 
eigenvectors are orthogonal by Property 3. Those eigenvectors are also real; they solve 
(A —AI)x =0. These orthonormal eigenvectors go into an orthogonal matrix Q, with 
O'Q =I and Q' = Q~!. Then S~!AS = A becomes special—it is Q~'AQ = A or 
A= QAQ~! = QAQ’. We can state one of the great theorems of linear algebra: 


50 Areal symmetric matrix can be factored into A= QAQ’. Its orthonormal 
eigenvectors are in the orthogonal matrix Q and its eigenvalues are in A. 


In geometry or mechanics, this is the principal axis theorem. It gives the right choice 
of axes for an ellipse. Those axes are perpendicular, and they point along the eigen- 
vectors of the corresponding matrix. (Section 6.2 connects symmetric matrices to n- 
dimensional ellipses.) In mechanics the eigenvectors give the principal directions, along 
which there is pure compression or pure tension—with no shear. 

In mathematics the formula A = QAQ! is known as the spectral theorem. If we 
multiply columns by rows, the matrix A becomes a combination of one-dimensional 
projections—which are the special matrices xx! of rank 1, multiplied by A: 


| || [An = i, 
A=QAQT= |x, ++ Xn 
| | An oq x, a 
= Ayxyxd + Agxgad +--+ Anryxl. 


Our 2 by 2 example has eigenvalues 3 and 1: 


(10) 


es | 1 _1 1 1 
Example 3. A= ee lee 2, 2 | + ]% 7] =combination of two projections. 
a ae > 2 


The eigenvectors, with length scaled to 1, are 


1 1 d 1 }1 
xX, =-> an XS = : 
see a | J/2/1 
Then the matrices on the right-hand side are ein) and x2x4—columns times rows—and 
they are projections onto the line through x, and the line through x2. 


All symmetric matrices are combinations of one-dimensional projections—which are 
symmetric matrices of rank 1. 
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Remark. If A is real and its eigenvalues happen to be real, then its eigenvectors are also 
real. They solve (A — AJ)x = 0 and can be computed by elimination. But they will not 
be orthogonal unless A is symmetric: A = QAQ! leads to AT = A. 

If A is real, all complex eigenvalues come in conjugate pairs: Ax = Ax and AX = AX. 
If a+ib is an eigenvalue of a real matrix, so is a— ib. (If A = A™ then b = 0.) 


Strictly speaking, the spectral theorem A = QAQ! has been proved only when the 
eigenvalues of A are distinct. Then there are certainly n independent eigenvectors, and 
A can be safely diagonalized. Nevertheless it is true (see Section 5.6) that even with 
repeated eigenvalues, a symmetric matrix still has a complete set of orthonormal eigen- 
vectors. The extreme case is the identity matrix, which has A = | repeated n times—and 
no shortage of eigenvectors. 

To finish the complex case we need the analogue of a real orthogonal matrix—and you 
can guess what happens to the requirement Q'Q = J. The transpose will be replaced by 
the conjugate transpose. The condition will become U#U = I. The new letter U reflects 
the new name: A complex matrix with orthonormal columns is called a unitary matrix. 


Unitary Matrices 


May we propose two analogies? A Hermitian (or symmetric) matrix can be compared 
to a real number. A unitary (or orthogonal) matrix can be compared to a number on 
the unit circle—a complex number of absolute value 1. The A’s are real if A! = A, and 
they are on the unit circle if U"U = J. The eigenvectors can be scaled to unit length and 
made orthonormal.® 

Those statements are not yet proved for unitary (including orthogonal) matrices. 
Therefore we go directly to the three properties of U that correspond to the earlier Prop- 
erties 1-3 of A. Remember that U has orthonormal columns: 


Unitary matrix U4U =1, UUH=7, and UF =uUH!. 


This leads directly to Property 1’, that multiplication by U has no effect on inner prod- 
ucts, angles, or lengths. The proof is on one line, just as it was for Q: 


Property 1! (Ux)#(Uy) = x#U4Uy = xy and lengths are preserved by U: 
Length unchanged \|U7x||? = x42u Fux = ||x\|?. (11) 
Property 2’ Every eigenvalue of U has absolute value |A| = 1. 


This follows directly from Ux = Ax, by comparing the lengths of the two sides: 
||Ux|| = ||x|| by Property 1’, and always ||A.x|| = |A|||x||. Therefore |A| = 1. 


6Later we compare “‘skew-Hermitian” matrices with pure imaginary numbers, and “normal” matrices with all 
complex numbers a+ ib. A nonnormal matrix without orthogonal eigenvectors belongs to none of these classes, 
and is outside the whole analogy. 
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Property 3’ Eigenvectors corresponding to different eigenvalues are orthonor- 
mal. 


Start with Ux = A,x and Uy = Ayy, and take inner products by Property 1’: 
xtly = (Ux)"(Uy) = (Arx)" (ay) = A daxiy. 


Comparing the left to the right, A; Az = 1 or x4y =0. But Property 2’ is AA; = 1, so we 
cannot also have A; A> = 1. Thus xHy = (0) and the eigenvectors are orthogonal. 
cost —sint 2 


Example 4. U = has eigenvalues e” and e 


sint cost 
The orthogonal eigenvectors are x = (1, —i) and y = (1,7). (Remember to take conjugates 
in xy = 147? =0.) After division by V2 they are orthonormal. 


Here is the most important unitary matrix by far. 


1 1 
Example 5. U = 1 ]l wo- we! _ Fourier matrix 
P ‘ a Jn . : . ; _ vn 
1 wl . wut)? 


The complex number w is on the unit circle at the angle @ = 27/n. It equals e?*/". Its 
powers are spaced evenly around the circle. That spacing assures that the sum of all n 
powers of w—all the nth roots of 1—is zero. Algebraically, the sum 1+ w+----+w"7! 
is (w” —1)/(w—1). And w” — 1 is zero! 


1 | 
row 1 of UF times column 2 of U is —(1+-w+w2+---+w"!) =" 7 
n w- 
Hy: a4 A 2 n—1,_ W"—1 
row i of U™ times column j of U is —(1+W+W°+---+W”")= Weal =, 
n — 


In the second case, W = w/~‘. Every entry of the original F has absolute value 1. The 
factor ,/n shrinks the columns of U into unit vectors. The fundamental identity of the 
finite Fourier transform is U"U = 1. 

Thus U is a unitary matrix. Its inverse looks the same except that w is replaced by 
w ! = e—!® — Wp. Since U is unitary, its inverse is found by transposing (which changes 
nothing) and conjugating (which changes w to W). The inverse of this U is U. Ux can be 


computed quickly by the Fast Fourier Transform as found in Section 3.5. 


By Property 1’ of unitary matrices, the length of a vector x is the same as the length 
of Ux. The energy in state space equals the energy in transform space. The energy is 
the sum of |x; >, and it is also the sum of the energies in the separate frequencies. The 
vector x = (1,0,...,0) contains equal amounts of every frequency component, and its 
Discrete Fourier Transform Ux = (1,1,...,1)/./n also has length 1. 
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Example 6. 


This is an orthogonal matrix, so by Property 3’ it must have orthogonal eigenvectors. 
They are the columns of the Fourier matrix! Its eigenvalues must have absolute value 1. 
They are the numbers 1,w,...,w”~! (or 1,i,i7,i in this 4 by 4 ease). It is a real matrix, 
but its eigenvalues and eigenvectors are complex. 


One final note, Skew-Hermitian matrices satisfy K' = —K, just as skew-symmetric 
matrices satisfy K' = —K. Their properties follow immediately from their close link to 
Hermitian matrices: 


If A is Hermitian then K = iA is skew-Hermitian. 


The eigenvalues of K are purely imaginary instead of purely real; we multiply i. The 
eigenvectors are not changed. The Hermitian example on the previous pages would lead 
to 

2i a+ SI 
—3-31~ St 


The diagonal entries are multiples of i (allowing zero). The eigenvalues are 8/7 and —i. 
The eigenvectors are still orthogonal, and we still have K = UAU"—with a unitary U 
instead of a real orthogonal Q, and with 87 and —i on the diagonal of A. 

This section is summarized by a table of parallels between real and complex. 


K=iA= = —K", 


Real versus Complex 
R" (n real components) o> C” (n complex components) 


length: ||x||? =x7+---+22 — length: |||]? = |xy|? + --- + |xnl? 
transpose: Al, = Aji ao Hermitian transpose: Aji = Ajj 
(AB)? =B'a! oO (AB)4 — BHAH 
inner product: xly =xXyp+-:-+Xyn <> inner product: xy =A ae an 
(Ax)"y =x" (ATy) = (Ax)Fy = xF(AMy) 
orthogonality: x'y = 0 o orthogonality: x4y = 0 
symmetric matrices: A! = A o Hermitian matrices: A# = A 
A=QAQ7! = QAO’ (teal A) o A =UAU~! =UAUH (teal A) 
skew-symmetric K' = —K > skew-Hermitian K4 = —K 
orthogonal 0'Q = or Q' = Qu! o unitary UHU = 7 or UH = U7! 
(Qx)"(Qy) =x"y and ||Ox|| = lx] (Ux) y) = x4 and ||U |] = ||| 


The columns, rows, and eigenvectors of Q and U are orthonormal, and every |A| = 1 
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Problem Set 5.5 


1, 


For the complex numbers 3 + 47 and 1 — i, 

(a) find their positions in the complex plane. 

(b) find their sum and product. 

(c) find their conjugates and their absolute values. 

Do the original numbers lie inside or outside the unit circle? 
What can you say about 

(a) the sum of a complex number and its conjugate? 

(b) the conjugate of a number on the unit circle? 

(c) the product of two numbers on the unit circle? 


(d) the sum of two numbers on the unit circle? 


If x =2+iand y= 1 +3), find xX, xx, 1/x, and x/y. Check that the absolute value |xy| 
equals |x| times |y|, and the absolute value |1/x| equals 1 divided by |.|. 


Find a and b for the complex numbers a-+ ib at the angles 6 = 30°,60°,90° on the 
unit circle. Verify by direct multiplication that the square of the first is the second, 
and the cube of the first is the third. 

2 
1 


(a) If x = re’? what are x 
numbers that have x~ 


,x_|, and ¥ in polar coordinates? Where are the complex 
=x 


(b) Att =0, the complex number e‘—!+#)! equals one. Sketch its path in the complex 
plane as ¢ increases from 0 to 27. 


Find the lengths and the inner product of 


2—4i 2+4i 
x= d y= 
Ai Ai 
Write out the matrix AM and compute C = AMA if 
aa fle) 
a | oa | 


What is the relation between C and CH? Does it hold whenever C is constructed from 
some AHA? 


(a) With the preceding A, use elimination to solve Ax = 0. 


(b) Show that the nullspace you just computed is orthogonal to C(A®) and not to 
the usual row space C(A'). The four fundamental spaces in the complex case 
are N(A) and C(A) as before, and then N(A#) and C(A®),. 
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2; 


10. 


11. 


12. 


13. 


14. 


15. 


16. 
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(a) How is the determinant of A# related to the determinant of A? 


(b) Prove that the determinant of any Hermitian matrix is real. 


(a) How many degrees of freedom are there in a real symmetric matrix, a real diag- 
onal matrix, and a real orthogonal matrix? (The first answer is the sum of the 
other two, because A = QAQ'.) 


(b) Show that 3 by 3 Hermitian matrices A and also unitary U have 9 real degrees of 
freedom (columns of U can be multiplied by any e’®). 


Write P, QO and R in the form Ayxyxl! + oxox of the spectral theorem: 


ae ee | 


Give a reason if true or a counterexample if false: 


P= 


NIE NIE 
NIP NI 


(a) If A is Hermitian, then A + iJ is invertible. 

(b) If Q is orthogonal. then Q + 51 is invertible. 

(c) If A is real, then A + i/ is invertible. 

Suppose A is a symmetric 3 by 3 matrix with eigenvalues 0, 1, 2. 

(a) What properties can be guaranteed for the corresponding unit eigenvectors u, v, 
w? 

(b) In terms of u, v, w, describe the nullspace, left nullspace, row space and column 
space of A. 

(c) Find a vector x that satisfies Ax = v-+ w. Is x unique? 

(d) Under what conditions on b does Ax = b have a solution? 


(e) If u, v, w are the columns of S, what are S~! and S~'AS? 


In the list below, which classes of matrices contain A and which contain B? 
: 
1 
i|- 
1 


Orthogonal, invertible, projection, permutation, Hermitian, rank-1, diagonalizable, 
Markov. Find the eigenvalues of A and B. 


pee) 
5 
Q 
& 
II 
| 
a 
a 
a 


What is the dimension of the space S of all n by n real symmetric matrices? The 
spectral theorem says that every symmetric matrix is a combination of n projection 
matrices. Since the dimension exceeds n, how is this difference explained? 


Write one significant fact about the eigenvalues of each of the following. 


17. 
18. 


19. 


20. 


21. 


22. 


23. 


24. 


25. 


26. 
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(a) A real symmetric matrix. 

(b) A stable matrix: all solutions to du/dt = Au approach zero. 
(c) An orthogonal matrix. 

(d) A Markov matrix. 

(e) A defective matrix (nondiagonalizable). 


(f) A singular matrix. 
Show that if U and V are unitary, so is UV. Use the criterion UHU = J. 


Show that a unitary matrix has |detU| = 1, but possibly detU is different from 
detU™. Describe all 2 by 2 matrices that are unitary. 


Find a third column so that U is unitary. How much freedom in column 3? 


1/V3 i/V2 
U=|1//3 0 
i/V3 1/v2 


Diagonalize the 2 by 2 skew-Hermitian matrix K = |], whose entries are all /—1. 
Compute e“’ = SeS~!, and verify that e*’ is unitary. What is the derivative of eX’ 
att = 0? 


Describe all 3 by 3 matrices that are simultaneously Hermitian, unitary, and diagonal. 
How many are there? 


Every matrix Z can be split into a Hermitian and a skew-Hermitian part, Z=A+ K, 
just as a complex number z is split into a+ ib, The real part of z is half of z+ z, and 
the “real part” of Z is half of Z+ Z". Find a similar formula for the “imaginary part” 
K, and split these matrices into A + K: 


347 442i 
0 2) 


and ZL = 


Show that the columns of the 4 by 4 Fourier matrix F in Example 5 are eigenvectors 
of the permutation matrix P in Example 6. 


For the permutation of Example 6, write out the circulant matrix C = col + ciP + 
CoP? + c3P>. (Its eigenvector matrix is again the Fourier matrix.) Write out also 
the four components of the matrix-vector product Cx, which is the convolution of 
c = (co,C1,C€2,¢3) and x = (x0, %1,%2,X3). 


For a circulant C= FAF~!, why is it faster to multiply by F~!, then A, then F (the 
convolution rule), than to multiply directly by C? 


Find the lengths of u = (1 +i,1—i,1+ 23) and v = (i,i,i). Also find w"v and vHu. 
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27. Prove that AMA is always a Hermitian matrix, Compute AMA and AA#: 


28. If Az = 0, then AHAz = 0. If AHAz = 0, multiply by z# to prove that Az = 0. The 
nullspaces of A and AMA are . AHA is an invertible Hermitian matrix when the 
nullspace of A contains only z = 


29. When you multiply a Hermitian matrix by a real number c, is cA still Hermitian? If 
c = 1, show that iA is skew-Hermitian. The 3 by 3 Hermitian matrices are a subspace, 
provided that the “scalars” are real numbers. 


30. Which classes of matrices does P belong to: orthogonal, invertible, Hermitian, uni- 
tary, factorizable into LU, factorizable into QR? 


31. Compute P’, P?, and P! in Problem 30. What are the eigenvalues of P? 


32. Find the unit eigenvectors of P in Problem 30, and put them into the columns of a 
unitary matrix U. What property of P makes these eigenvectors orthogonal? 


33. Write down the 3 by 3 circulant matrix C = 21+5P+4P?. It has the same eigen- 
vectors as P in Problem 30. Find its eigenvalues. 


34. If U is unitary and Q is a real orthogonal matrix, show that U~! is unitary and also 
UQ is unitary. Start from UU = J and O'O = 1. 


35. Diagonalize A (real A’s) and K (imaginary A’s) to reach UAU#: 
0 1-i 
= 
i+1 1 


36. Diagonalize this orthogonal matrix to reach Q = UAU#. Now all A’s are ; 


Ola 
1+i i 


_ cos@ —sin@ 
~ |sin@ cos@ 


37. Diagonalize this unitary matrix V to reach V = UAU". Again all |A| = 1: 


1 
V=— 
V3 


| ee os 
1+i —-l 


38. 


39. 


40. 
41. 
42. 
43. 


44. 
45. 


46. 


47. 
48. 
49. 


50. 
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If vj,...,V¥, is an orthonormal basis for C”, the matrix with those columns is a 
matrix. Show that any vector z equals (vEz)yy feet (VEZ) yy. 


The functions e~ and e~ are orthogonal on the interval 0 < x < 27 because their 
complex inner product is {57 =), 


The vectors v = (1,7,1), w= (i, 1,0) and z= are an orthogonal basis for ; 
If A= R+iS is a Hermitian matrix, are the real matrices R and S symmetric? 
The (complex) dimension of C” is . Find a nonreal basis for C”. 


Describe all 1 by 1 matrices that are Hermitian and also unitary. Do the same for 2 
by 2 matrices. 


How are the eigenvalues of A (square matrix) related to the eigenvalues of A? 


If w4u = 1, show that J — 2uu4 is Hermitian and also unitary. The rank-1 matrix wu! 


is the projection onto what line in C”? 


If A+B is a unitary matrix (A and B are real), show that Q = E re | is an orthogonal 


matrix. 


If A+ iB is a Hermitian matrix (A and B are real), show that | 5] is symmetric. 


Prove that the inverse of a Hermitian matrix is again a Hermitian matrix. 


Diagonalize this matrix by constructing its eigenvalue matrix A and its eigenvector 


matrix S: 
2 1-i 
1+i 3 


— AH. 


A matrix with orthonormal eigenvectors has the form A = UAU~! = UAU". Prove 
that AA! = AHA. These are exactly the normal matrices. 


5.6 Similarity Transformations 


Virtually every step in this chapter has involved the combination S~'AS. The eigenvec- 
tors of A went into the columns of S, and that made S~!AS a diagonal matrix (called 
A). When A was symmetric, we wrote Q instead of S, choosing the eigenvectors to be 
orthonormal. In the complex case, when A is Hermitian we write U—it is still the matrix 
of eigenvectors. Now we look at all combinations M~'A4AM—formed with any invertible 
M on the right and its inverse on the left. The invertible eigenvector matrix S may fail to 
exist (the defective case), or we may not know it, or we may not want to use it. 


First a new word: The matrices A and M—'AM are “similar”. Going from one to 


the other is a similarity transformation. It is the natural step for differential equations 
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or matrix powers or eigenvalues—just as elimination steps were natural for Ax = b. 
Elimination multiplied A on the left by L~!, but not on the right by L. So U is not 
similar to A, and the pivots are not the eigenvalues. 

A whole family of matrices M~'AM is similar to A, and there are two questions: 


1. What do these similar matrices M~!AM have in common? 


2. With a special choice of M, what special form can be achieved by M~'AM? 


The final answer is given by the Jordan form, with which the chapter ends. 
These combinations M~'AM arise in a differential or difference equation, when a 
“change of variables” u = Mv introduces the new unknown v: 
du dv 


d 
—=Au becomes M “Y= AM vy, or —=M~'!AMv 
dt dt dt 


Unt+1 =Au, becomes Myyi;=AMyy, or Vat = M~!AMyy. 


The new matrix in the equation is M~'AM. In the special case M = S, the system is 
uncoupled because A = S~!AS is diagonal. The eigenvectors evolve independently. This 
is the maximum simplification, but other M’s are also useful. We try to make M~'AM 
easier to work with than A. 

The family of matrices M~'AM includes A itself, by choosing M = J. Any of these 
similar matrices can appear in the differential and difference equations, by the change 
u = My, so they ought to have something in common, and they do: Similar matrices 
share the same eigenvalues. 


5P Suppose that B= M~'AM. Then A and B have the same eigenvalues. 
Every eigenvector x of A corresponds to an eigenvector M~'!x of B. 


Start from Ax = Ax and substitute A = MBM7—!: 
Same eigenvaluc MBM'x=Ax whichis B(M~'x)=A(M™!x). (1) 


The eigenvalue of B is still A. The eigenvector has changed from x to M~'!x. 
We can also check that A — AJ and B— AJ have the same determinant: 


Product of matrices B-—AI=M~'AM—AI=M~'(A—AIM 
Product rule det(B— AI) = detM~! det(A — AJ) detM = det(A — AJ). 


The polynomials det(A — A/) and det(B — AJ) are equal. Their roots—the eigenvalues 
of A and B—are the same. Here are matrices B similar to A. 
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Example 1. A = [} 9] has eigenvalues 1 and 0. Each B is M~'AM: 


1 1 
If M= e , then. B= 4 : triangular with A = 0 and 0. 
0 1 0 0 


If M= ; ac 2 
—1 1 


If M= : , then B = an arbitrary matrix with A = 0 and 0. 
C 


| > projection with A = 0 and 0. 


NIP NIE 
NIP NIE 


In this case we can produce any B that has the correct eigenvalues. It is an easy case, 
because the eigenvalues | and 0 are distinct. The diagonal A was actually A, the out- 
standing member of this family of similar matrices (the capo). The Jordan form will 
worry about repeated eigenvalues and a possible shortage of eigenvectors. All we say 
no is that every M~'AM has the same number of independent eigenvectors as A (each 
eigenvector is multiplied by M—!). 

The first step is to look at the linear transformations that lie behind the matrices. 
Rotations, reflections, and projections act on n-dimensional space. The transformation 
can happen without linear algebra, but linear algebra turns it into matrix multiplication. 


Change of Basis = Similarity Transformation 


The similar matrix B = M~'AM is closely connected to A, if we go back to linear trans- 
formations. Remember the key idea: Every linear transformation is represented by a 
matrix. The matrix depends on the choice of basis! If we change the basis by M we 
change the matrix A to a similar matrix B. 

Similar matrices represent the same transformation T with respect so different 
bases. The algebra is almost straightforward. Suppose we have a basis v1,...,Vn. The 
jth column of A comes from applying T to v;: 


Tv; = combination of the basis vectors = ay jvj +++: +njVn- (2) 


For a new basis V;,...,Vn, the new matrix B is constructed in the same way: TV; = 
combination of the V’s = bj ;V; +---+bnjVn. But also each V must be a combination 
of the old basis vectors: V; = )'mj;;v;. That matrix M is really representing the identity 
transformation (!) when the only thing happening is the change of basis (T is /). The in- 
verse matrix M~! also represents the identity transformation. when the basis is changed 
from the v’s back to the V’s. Now the product rule gives the result we want: 


5Q_ The matrices A and B that represent the same linear transformation T 
with respect to two different bases (the v’s and the V’s) are similar: 


[T]v tov = ZT] tov alr, T]v tov 


3 
B = M"' A M. 3) 
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I think an example is the best way to explain B = M~!AM. Suppose T is projection 
onto the line L at angle 9. This linear transformation is completely described without the 
help of a basis. But to represent T by a matrix, we do need a basis. Figure 5.5 offers two 
choices, the standard basis vj = (1,0), v2 = (0,1) and a basis V;, V2 chosen especially 
for T. 


135 3 = Sb WA 0 7 
\, 34 1 135°. Y= —@ it 
‘. os \ Y= 1 
ee —.5]*\ ‘\ 
projection 5 ae xy projects to zero 
. N N 
Ne 1 aN 
s wv {0 
are y 
oe V, 1 
5-5 S 1 0 aa 
A= | projection | | A= | 1 
—5 5 —5 00 projects to Vi 


Figure 5.5: Change of basis to make the projection matrix diagonal. 


In fact TV; = V, (since V, is already on the line L) and TV> = 0 (since V2 is perpen- 
dicular to the line). In that eigenvector basis, the matrix is diagonal: 


1 O 
Elgenvector basis B= ier= ; ; 


The other thing is the change of basis matrix M. For that we express V; as a combination 
vi;cos@1Lv2sin@ and put those coefficients into column 1. Similarly V2 (or /V2, the 
transformation is the identity) is —v; sin@ + v2 cos @, producing column 2: 


Change of basis M= veev= : a 
c - 


The inverse matrix M~! (which is here the transpose) goes from v to V. Combined with 
B and M, it gives the projection matrix in the standard basis of v’s: 


2 


Standard basis A — MBM! = : dl ; 
cs Ss 


We can summarize the main point. The way to simplify that matrix A—in fact to diag- 
onalize it—is to find its eigenvectors. They go into the columns of M (or S) and M~'!AM 
is diagonal. The algebraist says the same thing in the language of linear transformations: 
Choose a basis consisting of eigenvectors. The standard basis led to A, which was not 
simple. The right basis led to B, which was diagonal. 

We emphasize again that M~'AM does not arise in solving Ax = b. There the basic 
operation was to multiply A (on the left side only!) by a matrix that subtracts a multiple 
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of one row from another. Such a transformation preserved the nullspace and row space 
of A; it normally changes the eigenvalues. 

Eigenvalues are actually calculated by a sequence of simple similarities. The matrix 
goes gradually toward a triangular form, and the eigenvalues gradually appear on the 
main diagonal. (Such a sequence is described in Chapter 7.) This is much better than 
trying to compute det(A — AJ), whose roots should be the eigenvalues. For a large matrix, 
it is numerically impossible to concentrate all that information into the polynomial and 
then get it out again. 


Triangular Forms with a Unitary / 


Our first move beyond the eigenvector matrix M = S is a little bit crazy: Instead of a 
more general M, we go the other way and restrict M to be unitary. M~'AM can achieve 
a triangular form 7 under this restriction. The columns of M = U are orthonormal (in 
the real case, we would write M = Q). Unless the eigenvectors of A are orthogonal, a 
diagonal U~!AU is impossible. But “Schur’s lemma” in 5R is very useful—at least to 
the theory. (The rest of this chapter is devoted more to theory than to applications. The 
Jordan form is independent of this triangular form.) 


5R_ There is a unitary matrix M =U such that U~'AU = T is triangular. 


The eigenvalues of A appear along the diagonal of this similar matrix T. 


Proof. Every matrix, say 4 by 4, has at least one eigenvalue A;. In the worst case, it 
could be repeated four times. Therefore A has at least one unit eigenvector x;, which we 
place in the first column of U. At this stage the other three columns are impossible to 
determine, so we complete the matrix in any way that leaves it unitary, and call it U;. 
(The Gram-Schmidt process guarantees that this can be done.) Ax, = A,x,; column 1 
means that the product U, "AU 1 Starts in the right form: 


A, * * x A, * * x 

QO *« * x “1 * Ok Ok 
AU, =U, leads to U, AU; = 

O * * x* *k * x 

QO * * x* QO * * x 


Now work with the 3 by 3 submatrix in the lower right-hand corner. It has a unit 
eigenvector x2, which becomes the first column of a unitary matrix M: 


f 0 0 0 AL kK OK Ox 
Ax * OO 

If = h StU 'A = 
U> 0 Mp then U, (U; U,)U2 0 ae 
0 0 0 * x 
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At the last step, an eigenvector of the 2 by 2 matrix in the lower right-hand corner goes 
into a unitary M3, which is put into the corner of U3: 


Aq; Be oe 
0 A 
Triangular — Uy! (Uy 'U;!AU\U>) U3 = 2 eT 
0 Ax x 
0 0 O x 


The product U = U,U2U;3 is still a unitary matrix, and U “AU =T. 


This lemma applies to all matrices, with no assumption that A is diagoalizable. We 
could use it to prove that the powers A‘ approach zero when all \A;| < 1, and the expo- 
nentials e“' approach zero when all Red; < 0—even without the full set of eigenvectors 
which was assumed in Sections 5.3 and 5.4. 


Example 2. A = i 


The only line of eigenvectors goes through (1,1). After dividing by V2, this is the first 
column of U, and the triangular U~!AU = T has the eigenvalues on its diagonal: 


1/2 en 2 FA hee WKal= i by 


—] 
0 | has the eigenvalue A = 1 (twice). 


aie i es VO GU mee a 


Diagonalizing Symmetric and Hermitian Matrices 


This triangular form will show that any symmetric or Hermitian matrix—whether its 
eigenvalues are distinct or not—has a complete set of orthonormal eigenvectors. We 
need a unitary matrix such that U~'AU is diagonal. Schur’s lemma has just found it. 
This triangular T must be diagonal, because it is also Hermitian when A = A": 
T=T#® (vu tav)®=uBARo (A =u-lav. 
The diagonal matrix U~!AU represents a key theorem in linear algebra. 
5S (Spectral Theorem) Every real symmetric A can be diagonalized by an 
orthogonal matrix Q. Every Hermitian matrix can be diagonalized by a unitary 
U: 
(real) QO 'AQ=A or A=QAQ! 
(complex) U~'AU=A or A=UAU# 
The columns of Q (or U) contain orthonormal eigenvectors of A. 


Remark 1. In the real symmetric case, the eigenvalues and eigenvectors are real at every 
step. That produces a real unitary U—an orthogonal matrix. 
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Remark 2. A is the limit of symmetric matrices with distinct eigenvalues. As the limit 
approaches, the eigenvectors stay perpendicular. This can fail if A 4 A!: 


1 0 
A(@) = : sas 2 has eigenvectors and me ’ 
0 sind 0 sin 8 


As @ — 0, the only eigenvector of the nondiagonalizable matrix [9 4] is | }]. 
Example 3. The spectral theorem says that this A = A! can be diagonalized: 


0 1 0 
A= {1 0 O|_ withrepeated eigenvalues A, = Ap) =1 and A3 = —1. 
0 0 1 


A = | has a plane of eigenvectors, and we pick an orthonormal pair x; and x2: 


1 1 
: 1 d : d : 1 for A 1 
6 (ee an x2= an x3= = I] or =-l. 
1 7p ; 2 3 a : 3 


These are the columns of Q. Splitting A = QAQ! into 3 columns times 3 rows gives 


01 0 5 5 (0 000 5 -45 0 
A=11 0 0} =A |5 5 Of +4210 0 O] +431-3 5 0 
001 00 0 001 0 0 0 


Since A; = Ay, those first two projections Bae and xxh (each of rank 1) combine to give 
a projection P; of rank 2 (onto the plane of eigenvectors). Then A is 


010 i 30 5 35 0 
1 0 0] =APi+43P3=(+1)]/4 5 0] +(-1) }-5 4 0 (5) 
001 001 0 0 0 


Every Hermitian matrix with k different eigenvalues has a spectral decomposition into 
A=A\P,\+---+A,P,, where P; is the projection onto the eigenspace for A;. Since there is 
a full set of eigenvectors, the projections add up to the identity. And since the eigenspace 
are orthogonal, two projections produce zero: P;P; = 0. 


We are very close to answering an important question, so we keep going: For which 
matrices is T = A? Symmetric, skew-symmetric, and orthogonal 7’s are all diagonal! 
Hermitian, skew-Hermitian, and unitary matrices are also in this class. They correspond 
to numbers on the real axis, the imaginary axis, and the unit circle. Now we want the 
whole class, corresponding to all complex numbers. The matrices are called “normal”. 


5T The matrix N is normal if it commutes with N#!: NN# = NUN. For 
such matrices, and no others, the triangular T = U~'NU is the diagonal A. 
Normal matrices are exactly those that have a complete set of orthonormal 
eigenvectors. 
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Symmetric and Hermitian matrices are certainly normal: If A = AM, then AA! and 
A'A both equal A*. Orthogonal and unitary matrices are also normal: UU and U2U 
both equal J. Two steps will work for any normal matrix: 


1. If N is normal, then so is the triangular T = U~!NU: 
TT# = U—'NUUENYU = U-—'!NN¥U = U-!NENU = UPNFUU-'!NU = TET. 


2. A triangular T that is normal must be diagonal! (See Problems 19-20 at the end of 
this section.) 


Thus, if N is normal, the triangular T= U~!NU must be diagonal. Since T has the same 
eigenvalues as N, it must be A. The eigenvectors of N are the columns of U, and they 
are orthonormal. That is the good case. We turn now from the best possible matrices 
(normal) to the worst possible (defective). 

2s dl 

0 2| 


ae | 
Normal N | 1 i Defective A= 


The Jordan Form 


This section has done its best while requiring M to be a unitary matrix U. We got 
M~'AM into a triangular form T. Now we lift this restriction on M. Any matrix is 
allowed, and the goal is to make M~!AM as nearly diagonal as possible. 

The result of this supreme effort at diagonalization is the Jordan form J. If A has a full 
set of eigenvectors, we take M = S and arrive at J = S~'AS = A. Then the Jordan form 
coincides with the diagonal A. This is impossible for a defective (nondiagonalizable) 
matrix. For every missing eigenvector, the Jordan form will have a | just above its main 
diagonal. The eigenvalues appear on the diagonal because J is triangular. And distinct 
eigenvalues can always be decoupled. 

It is only a repeated A that may (or may not!) require an off-diagonal 1 in J. 


5U If A has s independent eigenvectors, it is similar to a matrix with s blocks: 


Jj 
Jordan form J=M7~'AM = me : (6) 
Js 


Each Jordan block J; is a triangular matrix that has only a single eigenvalue (; 
and only one eigenvector: 


Jordan block = ; (7) 
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The same A; will appear in several blocks, if it has several independent eigen- 
vectors. Two matrices are similar if and only if they share the same Jordan 
form J. 


Many authors have made this theorem the climax of their linear algebra course. 
Frankly, I think that is a mistake. It is certainly true that not all matrices are diagonaliz- 
able, and the Jordan form is the most general case. For that very reason, its construction 
is both technical and extremely unstable. (A slight change in A can put back all the 
missing eigenvectors, and remove the off-diagonal is.) Therefore the right place for the 
details is in the appendix, and the best way to start on the Jordan form is to look at some 
specific and manageable examples. 

Example 4. T = : i and A = a and B = ; : all lead to J = F i ; 
0 1 1 O 1 1 0 1 

These four matrices have eigenvalues | and | with only one eigenvector—so J con- 
sists of one block. We now check that. The determinants all equal 1. The traces (the 
sums down the main diagonal) are 2. The eigenvalues satisfy 1-1 = 1 and 1+ 1=2. For 
T, B, and J, which are triangular, the eigenvalues are on the diagonal. We want to show 
that these matrices are similar—they all belong to the same family. 


(T) From T to J, the job is to change 2 to 1. and a diagonal M will do it: 


pei EOF IL 2h Oli (PA) oe 
0 2} ]0 1] |0 5 0-4 


(B) From B to J, the job is to transpose the matrix. A permutation does that: 


Pall oto 12 


(A) From A to J, we go first to T as in equation (4). Then change 2 to 1: 


P-'BP= 


1 1 
U-'AU = a =T and then M'TM= = J. 
0 1 0 1 
0 1 2 00 1 
Example 5. A= |0 0 1 and B={|0 0 O}. 
00 0 00 0 


Zero is a triple eigenvalue for A and B, so it will appear in all their Jordan blocks. There 
can be a single 3 by 3 block, or a 2 by 2 and a 1 by I block, or three I by I blocks. Then 
A and B have three possible Jordan forms: 


010 0 
Ji.= 10 0 11, Jn = 10 
0 


0 0 
0 O;. (8) 
0 0 0 0 0 
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The only eigenvector of A is (1,0,0). Its Jordan form has only one block, and A must 
be similar to J;. The matrix B has the additional eigenvector (0,1,0), and its Jordan 
form is J2 with two blocks, As for J3 = zero matrix, it is in a family by itself; the only 
matrix similar to J3 is M~'0M = 0. A count of the eigenvectors will determine J when 
there is nothing more complicated than a triple eigenvalue. 


Example 6. Application to difference and differential equations (powers and expo- 
nentials). If A can be diagonalized, the powers of A = SAS! are easy: A = SA‘S~!. In 
every case we have Jordan’s similarity A = MJM~', so now we need the powers of J: 


A* = (MJM~')(MJM~').--(MJM~') = MJKM“!. 


J is block-diagonal, and the powers of each block can be taken separately: 


k 


A 10 AK KARL Sk(k—1)Ak-? 
YS 0) Heo S05 Ae kA} ; (9) 
000A 0 O AK 


This block J; will enter when A is a triple eigenvalue with a single eigenvector. Its 
exponential is in the solution to the corresponding differential equation: 


et t et 5 t2 ert 
Exponential e7”=| 0 e% te* |. (10) 
O20 eM 


Here J + Jit + (Jit)?/2!+-+- produces 1++At+A71?/2!+--- =e on the diagonal. 
The third column of this exponential comes directly from solving du/dt = Jju: 


d uj A 10 uy 0 
ap ee = OA 1] lw starting from uo = |0 
U3 0 0 Al lw 1 


This can be solved by back-substitution (since J; is triangular). The last equation du3/dt = 
Au3 yields uz = e*”. The equation for uz is dur /dt = Auz + us, and its solution is te™. 
The top equation is du, /dt = Au; + uo, and its solution is 5t7eM . When J has multi- 
plicity m with only one eigenvector, the extra factor t appears m— | times. 

These powers and exponentials of J are a part of the solutions uz and u(t). The other 


part is the M that connects the original A to the more convenient matrix J: 
if uy, =Au, then uz = Akuy = MIKM—!ug 
if du/dt=Au then u(t) = e“'u(0) =Me"M~!u(0). 
When M and J are S and A (the diagonalizable case) those are the formulas of Sections 


5.3 and 5.4. Appendix B returns to the nondiagonalizable case, and shows how the 
Jordan form can be reached. I hope the following table will be a convenient summary. 
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Similarity Transformations 
1. A is diagonalizable: The columns of S are eigenvectors and S~'AS = A. 


2. A is arbitrary: The columns of M include “generalized eigenvectors” of A, and the 
Jordan form M~'AM = J is block diagonal. 


3. A is arbitrary: The unitary U can be chosen so that U~!AU = T is triangular. 
4. A is normal, AA" = AMA: then U can be chosen so that U~!AU = A. 


Special cases of normal matrices, all with orthonormal eigenvectors: 
(a) If A = AF is Hermitian, then all A; are real. 

(b) If A =A! is real symmetric, then A is real and U = Q is orthogonal. 
(c) If A = —A# is skew-Hermitian, then all A; are purely imaginary. 


(d) If A is orthogonal or unitary, then all |A;| = 1 are on the unit circle. 


Problem Set 5.6 


1. If Bis similar to A and C is similar to B, show that C is similar to A. (Let B= M~'AM 
and C = N~'BN.) Which matrices are similar to [? 


2. Describe in words all matrices that are similar to |} °, |, and find two of them. 


3. Explain why A is never similar to A+ /. 
4. Find a diagonal M, made up of 1s and —1s, to show that 
2th 
A= : : is similar to B= 


= NO —_ 


1 

2 —1 2 
5. Show (if B is invertible) that BA is similar to AB. 

6. (a) If CD = —DC (and D is invertible), show that C is similar to —C. 


(b) Deduce that the eigenvalues of C must come in plus-minus pairs. 
(c) Show directly that if Cx = Ax, then C(Dx) = —A (Dx). 


7. Consider any A and a “Givens rotation” M in the 1-2 plane: 


abe cos@ —sin@ 0 
A=|de fl, M=sin@ cos@ 0O 
ghi 0 0 1 


Choose the rotation angle @ to produce zero in the (3,1) entry of M~'AM. 
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10. 


11. 


12. 


13. 


14. 


15. 


16. 
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Note. This “zeroing” is not so easy to continue, because the rotations that produce 
zero in place of d and h will spoil the new zero in the corner. We have to leave one 
diagonal below the main one, and finish the eigenvalue calculation in a different way. 
Otherwise, if we could make A diagonal and see its eigenvalues, we would be finding 
the roots of the polynomial det(A — A) by using only the square roots that determine 
cos @—and that is impossible. 


. What matrix M changes the basis V; = (1,1), V2 = (1,4) to the basis vy = (2,5), 


v2 = (1,4)? The columns of M come from expressing V; and V2 as combinations 
Lmijvi of the v’s. 


. For the same two bases, express the vector (3,9) as a combination c,V, + c2V2 and 


also as d,v, + dzv2. Check numerically that M connects c to d: Mc = d. 


Confirm the last exercise: If Vj = mjjvj +m1v2 and V2 = mj2vy + m22Vv2, and 
m41C, +my2C2 = d, and mz1c, +m22C2 = do, the vectors c,V, +c2V2 and djv; + d2v2 
are the same. This is the “change of basis formula” Mc = d. 


If the transformation T is a reflection across the 45° line in the plane, find its matrix 
with respect to the standard basis vj = (1,0), v2 = (0,1), and also with respect to 
V, = (1,1), Vo = (1,1). Show that those matrices are similar. 


The identity transformation takes every vector to itself: Tx = x. Find the corre- 
sponding matrix, if the first basis is v; = (1,2), v2 = (3,4) and the second basis is 
w, = (1,0), wz = (0,1). (It is not the identity matrix!) 


The derivative of a+ bx + cx” is b+ 2cx + 0x’. 
(a) Write the 3 by 3 matrix D such that 


(b) Compute D? and interpret the results in terms of derivatives. 


(c) What are the eigenvalues and eigenvectors of D? 


Show that every number is an eigenvalue for T f(x) = df/dx, but the transformation 
T f (x) = Jo f (t)dt has no eigenvalues (here —o° < x < ©). 


On the space of 2 by 2 matrices, let T be the transformation that transposes every 
matrix. Find the eigenvalues and “eigenmatrices” for AT = 2A. 


(a) Find an orthogonal Q so that O~-'AQ = A if 


1141 0 
A=1]1 11 and A= |0 
111 0 


17. 


18. 


19. 


20. 


21. 


22. 


23. 
24. 


25. 


26. 


27. 
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Then find a second pair of orthonormal eigenvectors x, x2 for A = 0. 
(b) Verify that P = ie +x2x3 is the same for both pairs. 
Prove that every unitary matrix A is diagonalizable, in two steps: 
(i) If A is unitary, and U is too, then sois T = U TAU 
(ii) An upper triangular T that is unitary must be diagonal. Thus T = A. 


Any unitary matrix A (distinct eigenvalues or not) has a complete set of orthonormal 
eigenvectors. All eigenvalues satisfy |A| = 1. 


Find a normal matrix (VN# = NEN) that is not Hermitian, skew-Hermitian, unitary, 
or diagonal. Show that all permutation matrices are normal. 


Suppose T is a 3 by 3 upper triangular matrix, with entries f;;. Compare the entries of 
TT" and T#7, and show that if they are equal, then 7 must be diagonal. All normal 
triangular matrices are diagonal. 


If N is normal, show that ||Nx|| = ||N"x|| for every vector x. Deduce that the ith row 
of N has the same length as the ith column. Note: If N is also upper triangular, this 
leads again to the conclusion that it must be diagonal. 


Prove that a matrix with orthonormal eigenvectors must be normal, as claimed in 5T: 
If U-'NU =A, or N =UAU", then NNE = NEN, 


Find a unitary U and triangular T so that U~'AU = T, for 


j- 23 
AS 2p 


0 1 0 
A= | and A=1|10 0 0 
100 


If A has eigenvalues 0, 1, 2, what are the eigenvalues of A(A —/)(A — 2)? 


(a) Show by direct multiplication that every triangular matrix T, say 3 by 3, satisfies 
its own characteristic equation: (T — A,/)(T — Agl)(T — Asl) = 0. 

(b) Substituting U~'AU for T, deduce the famous Cayley-Hamilton theorem: Every 
matrix satisfies its own characteristic equation. For 3 by 3 this is (A — A,J) (A — 
AzI)(A — A3I) = 0. 


The characteristic polynomial of A = [45] is A? — (a+d)A + (ad —bc). By direct 
substitution, verify Cayley-Hamilton: A? — (a+d)A + (ad —bc)I = 0. 


If aj; = 1 above the main diagonal and a;; = 0 elsewhere, find the Jordan form (say 
4 by 4) by finding all the eigenvectors. 


Show, by trying for an M and failing, that no two of the three Jordan forms in equa- 
tion (8) are similar: J) 4 M~'JM, J, 4 M~'J3M, and Jp AM! 3M. 
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28. Solve uw’! = Ju by back-substitution, solving first for u2(t): 


d 1 : 
nae ee e | "| with initial value (0) = | | : 
dt 0 : 


Notice te” in the first component u; (ft). 


29. Compute A!° and e4 if A= MJM7!: 


Leal lee 


30. Show that A and B are similar by finding M so that B= M~'AM: 


A= 


(a) A= : ; and B= ; i ‘ 
1 0 0 1 
(b) A= : | and B= | , | 
1 1 —1 1 
(c) A= i and B= ; ; 3 
3 4 pm | 


31. Which of these matrices A; to Ag are similar? Check their eigenvalues. 


eb bd bY bal bi 


32. There are sixteen 2 by 2 matrices whose entries are 0s and 1s. Similar matrices go 
into the same family. How many families? How many matrices (total 16) in each 
family? 


33. (a) Ifx is in the nullspace of A, show that M ~ly is in the nullspace of M~ VAM. 
(b) The nullspaces of A and M —lAM have the same (vectors)(basis)(dimension). 


34. IfA and B have the exactly the same eigenvalues and eigenvectors, does A = B? With 
n independent eigenvectors, we do have A = B. Find A # B when A = 0,0 (repeated), 
but there is only one line of eigenvectors (x;,0). 


Problems 35-39 are about the Jordan form. 


35. By direct multiplication, find J* and J? when 


Guess the form of J*. Set k = 0 to find J°. Set k = —1 to find J~!. 


36. 


37. 


38. 


39. 


40. 


41. 


42. 
43. 
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If J is the 5 by 5 Jordan block with A = 0, find J* and count its eigenvectors, and 
find its Jordan form (two blocks). 


The text solved du/dt = Ju for a 3 by 3 Jordan block J. Add a fourth equation 
dw /dt = 5w +x. Follow the pattern of solutions for z, y, x to find w. 


These Jordan matrices have eigenvalues 0, 0, 0, 0. They have two eigenvectors (find 
them). But the block sizes don’t match and J is not similar to K: 


0 1\;0 0 0 1 0|0 
gcei| 2 8100 rm x_|9 9 1/9 
00/0 1 00 0/0 
0 0'0 0 0 0 0|0 


For any matrix M, compare JM with MK. If they are equal, show that M is not 
invertible. Then M~'JM = K is impossible. 


Prove in three steps that A! is always similar to A (we know that the 2’s are the 
same, the eigenvectors are the problem): 


(a) For A = one block, find M; = permutation so that M, ‘TM; = J : 
(b) For A = any J, build Mo from blocks so that My 'TMo = Jt, 
(c) For any A= MJM ~!: Show that A! is similar to J? and so to J and to A. 


Which pairs are similar? Choose a, b, c, d to prove that the other pairs aren’t: 


ra bd bal bed 


True or false, with a good reason: 


(a) An invertible matrix can’t be similar to a singular matrix. 

(b) A symmetric matrix can’t be similar to a nonsymmetric matrix. 
(c) A can’t be similar to —A unless A = 0. 

(d) A—J can’t be similar to A+ J. 

Prove that AB has the same eigenvalues as BA. 


If A is 6 by 4 and B is 4 by 6, AB and BA have different sizes. Nevertheless, 


ITA 
0 1 


0 O 
B BA 


I -A 
0 TT 


AB 0 
B O 


=G. 


(a) What sizes are the blocks of G? They are the same in each matrix. 

(b) This equation is M -lFM =G,s0 F and G have the same 10 eigenvalues. F has 
the eigenvalues of AB plus 4 zeros; G has the eigenvalues of BA plus 6 zeros. AB 
has the same eigenvalues as BA plus zeros. 
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44. Why is each of these statements true? 


(a) If A is similar to B, then A? is similar to B?. 

(b) A? and B? can be similar when A and B are not similar (try A = 0,0). 
(c) [§ 4] is similar to [3 4]. 

(d) [33] is not similar to [3 3]. 


(e) If we exchange rows | and 2 of A, and then exchange columns | and 2, the 
eigenvalues stay the same. 


Properties of Eigenvalues and Eigenvectors 


How are the properties of a matrix reflected in its eigenvalues and eigenvectors? This 
question is fundamental throughout Chapter 5. A table that organizes the key facts may 
be helpful. For each class of matrices, here are the special properties of the eigenvalues 
A; and eigenvectors xj. 


Symmetric: A’ =A real A’s orthogonal x} x; = 0 
Orthogonal: Q' = Q7! all |A| = 1 orthogonal x} x; = 0 
Skew-symmetric: A! = —A imaginary A’s orthogonal cae j=O0 
Complex Hermitian: A’ =A teald’s orthogonal x}x; = 0 

Positive definite: x'Ax > 0 allA >0 orthogonal 

Similar matrix: B=M~!AM A(B) =A(A) x(B) =M~!x(A) 

Projection: P =P? = P! A=1;0 column space; nullspace 
Reflection: /—2uu' A=-1:1,...,1 usu- 

Rank-1 matrix: uv! A=v'u;0,...,0 usv~ 

Inverse: A~! 1/A(A) eigenvectors of A 

Shift: A+ cl A(A) +e eigenvectors of A 

Stable powers: A” — 0 all |A| <1 

Stable exponential: e470 all Rea <0 

Markov: mjj >0, Yj, mij=1 Amax = 1 steady state x > 0 

Cyclic permutation: P?=I A, = e2K/n pS Agsehe 
Diagonalizable: SAS! diagonal of A columns of S are independent 
Symmetric: QAQ' diagonal of A (real) columns of Q are orthonormal 
Jordan: J =M~'AM diagonal of J each block gives | eigenvector 


Every matrix: A =USV" rank(A) =rank(Z) eigenvectors of ATA, AA! in V, U 
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Review Exercises 
5.1 Find the eigenvalues and eigenvectors, and the diagonalizing matrix S, for 
1 0 7 2 
and B= ’ 
2 ; " i 15 | 


5.2 Find the determinants of A and A~! if 


A= 


5.3 If A has eigenvalues 0 and 1, corresponding to the eigenvectors 


1 2 
and ; 
how can you tell in advance that A is symmetric? What are its trace and determi- 


nant? What is A? 


5.4 In the previous problem, what will be the eigenvalues and eigenvectors of A*? What 
is the relation of A? to A? 


5.5 Does there exist a matrix A such that the entire family A + c/ is invertible for all 
complex numbers c? Find a real matrix with A + r/ invertible for all real r. 


5.6 Solve for both initial values and then find e“’: 


du |3 1 : 1 ; 0 
ee ; ‘ u if u(0)= 4 andif u(0) = ql 


5.7 Would you prefer to have interest compounded quarterly at 40% per year, or annu- 
ally at 50%? 


5.8 True or false (with counterexample if false): 


(a) If B is formed from A by exchanging two rows, then B is similar to A. 

(b) If a triangular matrix is similar to a diagonal matrix, it is already diagonal. 

(c) Any two of these statements imply the third: A is Hermitian, A is unitary, A? = /. 
(d) If A and B are diagonalizable, so is AB. 


5.9 What happens to the Fibonacci sequence if we go backward in time, and how is F_; 
related to F,? The law Fy42 = F4,+F; is still in force, so F_; = 1. 


5.10 Find the general solution to du/dt = Au if 
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5.11 


5.12 
5.13 


5.14 


3.15 


5.16 


5.17 


5.18 
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Can you find a time T at which the solution u(T) is guaranteed to return to the 
initial value u(0)? 


If P is the matrix that projects R” onto a subspace S, explain why every vector in 
S is an eigenvector, and so is every vector in St. What are the eigenvai (Note the 
connection to P? = P, which means that A? = A.) 


Show that every matrix of order > 1 is the sum of two singular matrices. 


(a) Show that the matrix differential equation dX /dt = AX + XB has the solution 
X(t) =e*"X (O)e*. 

(b) Prove that the solutions of dX /dt = AX — XA keep the same eigenvalues for all 
time. 


If the eigenvalues of A are | and 3 with eigenvectors (5,2) and (2,1), find the 
solutions to du/dt = Au and u;z,, = Aug, starting from u = (9,4). 


Find the eigenvalues and eigenvectors of 


0 —i O 
A=]i 1 i 
O00 


What property do you expect for the eigenvectors, and is it true? 


eee 


show that A has no square root. Change the diagonal entries of A to 4 and find a 
square root. 


By trying to solve 


=A 


(a) Find the eigenvalues and eigenvectors of A = E 0| : 
(b) Solve du/dt = Au starting from u(0) = (100, 100). 


(c) If v(t) = income to stockbrokers and w(t) = income to client, and they help 
each other by dv/dt = 4w and dw/dt = jv, what does the ratio v/w approach 
as t — oo? 


True or false, with reason if true and counterexample if false: 


(a) For every matrix A, there is a solution to du/dt = Au starting from u(0) = 
CLgehs 1) 

(b) Every invertible matrix can be diagonalized. 

(c) Every diagonalizable matrix can be inverted. 


(d) Exchanging the rows of a 2 by 2 matrix reverses the signs of its eigenvalues. 


5.19 


5.20 


5.21 


5.22 


5.23 
5.24 


5.25 


5.26 


5.27 
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(e) If eigenvectors x and y correspond to distinct eigenvalues, then x4y = 0. 


If K is a skew-symmetric matrix, show that Q = (J — K)(I+K)7~! is an orthogonal 
matrix. Find Q if K = ie ab 

If K# = —K (skew-Hermitian), the eigenvalues are imaginary and the eigenvectors 
are orthogonal. 

(a) How do you know that K —/ is invertible? 

(b) How do you know that K = UAU H fora unitary U? 

(c) Why is e™ unitary? 

(d) Why is eX’ unitary? 


If M is the diagonal matrix with entries d, d*, d*, what is M~'AM? What are its 
eigenvalues in the following case? 


111 
A={1 11 
jes | 
If A? = —J, what are the eigenvalues of A? If A is a real n by n matrix show that n 


must be even, and give an example. 
If Ax = Ayx and Aly = Apy (all real), show that x'y = 0. 
A variation on the Fourier matrix is the “sine matrix”: 


sin@ sin20 sin3@ - 

S = — |sin20 sin4@ sin6@ with @=-—. 

08h 4 
sin3@ sin6@ sin9@ 


Verify that S' = S~!. (The columns are the eigenvectors of the tridiagonal —1, 2, 

—1] matrix.) 

(a) Find a nonzero matrix N such that N? = 0. 

(b) If Nx = Ax, show that A must be zero. 

(c) Prove that N (called a “nilpotent” matrix) cannot be symmetric. 

(a) Find the matrix P = aa'/a'a that projects any vector onto the line through 
aS(251,2) 

(b) What is the only nonzero eigenvalue of P, and what is the corresponding eigen- 
vector? 


(c) Solve uz+1 = Pug, starting from up = (9,9,0). 


Suppose the first row of A is 7, 6 and its eigenvalues are i, —i. Find A. 
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5.28 (a) For which numbers c and d does A have real eigenvalues and orthogonal eigen- 
vectors? 

12. 0 

A=|2 de 

0. 3.3 


(b) For which c and d can we find three orthonormal vectors that are combinations 
of the columns (don’t do it!)? 


5.29 If the vectors x; and x2 are in the columns of S, what are the eigenvalues and eigen- 


vectors of 
2; 2 
‘ o> and B=S : slo 
0 1 0 1 


A=S 


A 
5.30 What is the limit as k — oo (the Markov steady state) of | 6 ; A ? 


Positive Definite Matrices 


6.1. Minima, Maxima, and Saddle Points 


Up to now, we have hardly thought about the signs of the eigenvalues. We couldn’t 
ask whether A was positive before it was known to be real. Chapter 5 established that 
every symmetric matrix has real eigenvalues. Now we will find a test that can be applied 
directly to A, without computing its eigenvalues, which will guarantee that all those 
eigenvalues are positive. The test brings together three of the most basic ideas in the 
book—pivots, determinants, and eigenvalues. 

The signs of the eigenvalues are often crucial. For stability in differential equations, 
we needed negative eigenvalues so that e”’ would decay. The new and highly important 
problem is to recognize a minimum point. This arises throughout science and engi- 
neering and every problem of optimization. The mathematical problem is to move the 
second derivative test F” > 0 into n dimensions. Here are two examples: 


F(x,y) =7+2(x+y)* —ysiny—x° f (x,y) = 2x" +4axy+y?. 


Does either F (x,y) or f(x,y) have a minimum at the point x = y = 0? 


Remark 3. The zero-order terms F (0,0) = 7 and f(0,0) = 0 have no effect on the an- 
swer. They simply raise or lower the graphs of F and f. 


Remark 4, The linear terms give a necessary condition: To have any chance of a mini- 
mum, the first derivatives must vanish at x = y = 0: 


OF OF 
— =4(x+y)—3x7 =0 and — =4(x+y)—ycosy—siny =0 


Ox oy 
0 
aay ee, and OT Ny ogy All zero. 
Ox oy 


Thus (x,y) = (0,0) is a stationary point for both functions. The surface z = F(x, y) is 
tangent to the horizontal plane z = 7, and the surface z = f(x,y) is tangent to the plane 
z=0. The question is whether the graphs go above those planes or not, as we move 
away from the tangency point x = y= 0. 
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Remark 5. The second derivatives at (0,0) are decisive: 


O°F of 
a Of Of _ 
Oxdy dydox — Oxdy  dydx — 
0°F O° f 

Wye eee a2 


These second derivatives 4, 4, 2 contain the answer. Since they are the same for F' and 
f, they must contain the same answer. The two functions behave in exactly the same 
way near the origin. F has a minimum if and only if f has a minimum. J am going to 
show that those functions don’t! 


Remark 6. The higher-degree terms in F have no effect on the question of a local min- 
imum, but they can prevent it from being a global minimum. In our example the term 
—x> must sooner or later pull F toward —co. For f(x,y), with no higher terms, all the 
action is at (0,0). 

Every quadratic form f = ax? + 2bxy+ cy? has a stationary point at the origin, where 
Of /dx = 0f/dy =0. A local minimum would also be a global minimum, The surface 
z= f(x,y) will then be shaped like a bowl, resting on the origin (Figure 6.1). If the 
stationary point of F is at x = a, y = B, the only change would be to use the second 
derivatives at a, B: 


Quadratic x° O°F O°F y OF 


(ator FY) =F Zap (OB) tay (a, B) + 5 He (8). WD 


This f(x,y) behaves near (0,0) in the same way that F(x, y) behaves near (a, 8). 


f= o ca y? 
Y 
x 
Figure 6.1: A bowl and a saddle: Definite A = Fi al and indefinite A = 5 ae 


The third derivatives are drawn into the problem when the second derivatives fail to 
give a definite decision. That happens when the quadratic part is singular. For a true 
minimum, f is allowed to vanish only at x = y = 0. When f(x,y) is strictly positive at 
all other points (the bowl! goes up), it is called positive definite. 
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Definite versus Indefinite: Bowl versus Saddle 


The problem comes down to this: For a function of two variables x and y, what is the 
correct replacement for the condition 07F /dx* > 0? With only one variable, the sign of 
the second derivative decides between a minimum or a maximum. Now we have three 
second derivatives: Fyx, Fy = Kx, and ,. These three numbers (like 4, 4, 2) must 
determine whether or not F (as well as f) has a minimum. 

What conditions on a, b, and c ensure that the quadratic f (x,y) = ax* + 2bxy +cy? 
is positive definite? One necessary condition is easy: 


(i) If ax* + 2bxy+ cy? is positive definite, then necessarily a > 0. 


We look at x = 1, y = 0, where ax* + 2bxy + cy* is equal to a. This must be positive. 
Translating back to F, that means that 0?F /dx* > 0. The graph must go up in the x 
direction. Similarly, fix x = 0 and look in the y direction where f(0,y) = cy”: 


(ii) If f(x,y) is positive definite, then necessarily c > 0. 


Do these conditions a > 0 and c > O guarantee that f(x,y) is always positive? The 
answer is no. A large cross term 2bxy can pull the graph below zero. 


Example 1. f(x,y) =x? — 10xy+y’. Here a = 1 and c = 1 are both positive. But f is 
not positive definite, because f(1,1) = —8. The conditions a > 0 and c > 0 ensure that 
f(x,y) is positive on the x and y axes. But this function is negative on the line x = y, 
because b = —10 overwhelms a and c. 


Example 2. In our original f the coefficient 2b = 4 was positive. Does this ensure a 
minimum? Again the answer is no; the sign of b is of no importance! Even though its 
second derivatives are positive, 2x” + 4xy + y? is not positive definite. Neither F nor f 
has a minimum at (0,0) because f(1,—1) =2—-—4+1=-1. 


It is the size of b, compared to a and c, that must be controlled. We now want a 
necessary and sufficient condition for positive definiteness. The simplest technique is to 
complete the square: 


2 2 

Express f(x, b b 
P Fy) f =a? +2bxy bey? =a(x+2y) Te (--=) are. 12) 

using squares a a 
The first term on the right is never negative, when the square is multiplied by a > 0. 
But this square can be zero, and the second term must then be positive. That term 
has coefficient (ac — b*)/a. The last requirement for positive definiteness is that this 


coefficient must be positive: 


(iii) If ax* + 2bxy + cy” stays positive, then necessarily ac > b?. 


Testforaminimum: The conditions a > 0 and ac > b’ are just right. They guarantee 
c > 0. The right side of (2) is positive, and we have found a minimum: 
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6A ax? +2bxy+cy’ is positive definite if and only if a > 0 and ac > b*. Any 
f(x,y) has a minimum at a point where OF /dx = OF /dy = 0 with 


OF? OF?) [aF?] _ [ aF?]° 
geet Me ella 


Test fora maximum: Since f has a maximum whenever — f has a minimum, we just 
reverse the signs of a, b, and c. This actually leaves ac > b* unchanged: The quadratic 
form is negative definite if and only if a < 0 and ac > b*. The same change applies for 
a maximum of F (x,y). 


Singular case ac =b*: The second term in equation (2) disappears to leave only 
the first square—which is either positive semidefinite, when a > 0, or negative semidef- 
inite, when a < (0. The prefix semi allows the possibility that f can equal zero, as it will 
at the point x = b, y = —a. The surface z = f(x, y) degenerates from a bow] into a valley. 
For f = (x+y), the valley runs along the line x+y = 0. 


Saddle Point ac < b?: In one dimension, F(x) has a minimum or a maximum, or 
F" = 0. In two dimensions, a very important possibility still remains: The combination 
ac — b* may be negative. This occurred in both examples, when b dominated a and c. It 
also occurs if a and c have opposite signs. Then two directions give opposite results—in 
one direction f increases, in the other it decreases. It is useful to consider two special 
cases: 


Saddle points at (0,0) =f; =2xy and fy=x?-y? and ac—b’=-1. 


In the first, b = 1 dominates a = c = 0. In the second, a = 1 and c = —1 have opposite 
sign. The saddles 2xy and x* — y’ are practically the same; if we turn one through 45° 
we get the other. They are also hard to draw. 

These quadratic forms are indefinite, because they can take either sign. So we have 
a stationary point that is neither a maximum or a minimum. It is called a saddle point. 
The surface z = x* — y* goes down in the direction of the y axis, where the legs fit (if you 
still ride a horse). In case you switched to a car, think of a road going over a mountain 
pass. The top of the pass is a minimum as you look along the range of mountains, but it 
is a Maximum as you go along the road. 


Higher Dimensions: Linear Algebra 


Calculus would be enough to find our conditions Fy, > 0 and Fy.Fyy > ES for a minimum. 
But linear algebra is ready to do more, because the second derivatives fit into a symmetric 
matrix A. The terms ax? and cy” appear on the diagonal. The cross derivative 2bxy is 
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split between the same entry b above and below. A quadratic f(x,y) comes directly from 
a symmetric 2 by 2 matrix! 


x! Ax in R? ax + 2bxy+cy* = E y| ; ; A (4) 
b c| ly 
This identity (please multiply it out) is the key to the whole chapter. It generalizes 
immediately to n dimensions, and it is a perfect shorthand for studying maxima and 
minima. When the variables are x,,...,X,, they go into a column vector x. For any 
symmetric matrix A, the product x' Ax is a pure quadratic form f (x,,...,Xn): 


ait 412 + Ain} |%1 
, a2} 422 + A2n| |XxX2 
xAx in R" Le xD - Xl =). Giwings © 13) 
e a. u ; ha 
Gni 4n2 * Ann Xn 


The diagonal entries aj; to Gn multiply 7 to x, The pair aj; = aj; combines into 


2a; ;xix;. Then f = ayixt + 2ajoxX1X2 +++ + Awe. 

There are no higher-order terms or lower-order terms—only second-order. The func- 
tion is zero at x = (0,...,0), and its first derivatives are zero. The tangent is flat; this is 
a stationary point. We have to decide if x = 0 is a minimum or a maximum or a saddle 
point of the function f = x'Ax. 


Example 3. f = 2x7 + 4xy+ y’ and A = 


2 
— saddle point. 


Example 4. f = 2xy and A = — saddle point. 


1 
0 


Example 5. A is 3 by 3 for 2x4 —2x1xo + 2G — 2x2x3 + 2x3 


2 -l1 O xX] 
(ie Le x2 x3, —1 2 —-1] |x! — minimum at (0,0,0). 
0 —1 2 X3 


Any function F(x1,...,X,) is approached in the same way. At a stationary point 
all first derivatives are zero. A is the “second derivative matrix” with entries a;j = 
0°F /dx;0x;. This automatically equals aj; = 0?F /Oxj0x;, so A is symmetric. Then F 
has a minimum when the pure quadratic x' Ax is positive definite. These second-order 
terms control F near the stationary point: 


1 
Taylor series F(x) = F(0) +x" (grad F) + at Ax +higher orderterms. (6) 


At a stationary point, grad F = (OF /0x,,...,0F /Ox,) is a vector of zeros. The second 
derivatives in x'Ax take the graph up or down (or saddle). If the stationary point is at xo 
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instead of 0, F(x) and all derivatives are computed at xo. Then x changes to x — x9 on 
the right-hand side. 

The next section contains the tests to decide whether x! Ax is positive (the bowl goes 
up from x = 0). Equivalently, the tests decide whether the matrix A is positive defi- 
nite—which is the main goal of the chapter. 


Problem Set 6.1 
1. The quadratic f = x7 +4xy+2y7 has a saddle point at the origin, despite the fact that 
its coefficients are positive. Write f as a difference of two squares. 


2. Decide for or against the positive definiteness of these matrices, and write out the 
corresponding f = x! Ax: 


13 j.. 24 2 3 2f..3 
(a) E Ar (b) fe at (c) ; Ar © E a) 


The determinant in (b) is zero; along what line is f(x,y) = 0? 


3. If a 2 by 2 symmetric matrix passes the tests a > 0, ac > b*, solve the quadratic 
equation det(A — AJ) = 0 and show that both eigenvalues are positive. 


4. Decide between a minimum, maximum, or saddle point for the following functions. 
(a) F = —14 4(e* — x) — 5xsiny + 6y” at the point x = y = 0. 
(b) F = (x* — 2x) cosy, with stationary point at x = 1, y= 2. 
5. (a) For which numbers b is the matrix A = |} $] positive definite? 
(b) Factor A = LDL! when b is in the range for positive definiteness. 
(c) Find the minimum value of 5 (x7 + 2bxy + 9y’) —y for b in this range. 
(d) What is the minimum if b = 3? 


6. Suppose the positive coefficients a and c dominate b in the sense that a+c > 2b. 
Find an example that has ac < b*, so the matrix is not positive definite. 


7. (a) What 3 by 3 symmetric matrices A; and A2 correspond to f; and f2? 


f= eI sa ae — 2x1 x2 — 2x1x3 + 2x2x3 
f= ie + 2g + 11x} — 2x 1X2 — 2x1x3 — 4x2Xx3. 


(b) Show that f; is a single perfect square and not positive definite. Where is /) 
equal to 0? 


(c) Factor A> into LL', Write f> = x'A>x as a sum of three squares. 
q 


8. If A = [4°] is positive definite, test A~' = [4 4] for positive definiteness. 
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9. The quadratic f(x1,x2) = 3(x; + 2x2)? + 4x5 is positive. Find its matrix A, factor it 
into LDL’, and connect the entries in D and L to 3, 2, 4 in f. 
10. If R=[f 4], write out R? and check that it is positive definite unless R is singular. 


11. (a) IfA = [¢°] is Hermitian (complex b), find its pivots and determinant. 


(b) Complete the square for x4 Ax. Now x4 = [x, x2] can be complex 
a\x, |? + 2RebX1x2 + c|x2|? = alxy + (b/a)xo|? +____|xo’. 

(c) Show that a > 0 and ac > |b|* ensure that A is positive definite. 

(d) Are the matrices | ,'; '}"] and [,°, “¢"] positive definite? 


12. Decide whether F = x*y? — 2x — 2y has a minimum at the point x = y = 1 (after 
showing that the first derivatives are zero at that point). 


13. Under what conditions on a, b, c is ax* + 2bxy+cy* > x? + y’ for all x, y? 
Problems 14-18 are about tests for positive definiteness. 


14. Which of Aj, A, A3, Ag has two positive eigenvalues? Test a > 0 and ac > b?, don’t 
compute the eigenvalues. Find an x so that x'A,x <0. 


al EZ he 1 10 oe: 1 10 . 
=.= 5 10 100 10 101 


15. What is the quadratic f = ax* +2bxy + cy’ for each of these matrices? Complete the 
q 


A, = A) = 


6 7 


square to write f as a sum of one or two squares d)( )? + do( ee 
t 2 1 
A= and A= : 
29 3 9 


16. Show that f(x,y) = x*+4xy+3y? does not have a minimum at (0,0) even though 
it has positive coefficients. Write f as a difference of squares and find a point (x,y) 
where f is negative. 


17. (Important) If A has independent columns, then A'A is square and symmetric and 
invertible (Section 4.2). Rewrite x'A'Ax to show why it is positive except when 
x = 0. Then A!A is positive definite. 


18. Test to see if A‘A is positive definite in each case: 


A= 


5 1 1 
: A=1]1 2], and A= 
U3 


od, 2 
2 ay 
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19. Find the 3 by 3 matrix A and its pivots, rank, eigenvalues, and determinant: 


x] 
x1 x2 5 A x2 = A(x, — x2 +2x3)?. 
X3 


20. For F\(x,y) = ixt +x°y+y* and Fy(x,y) = x° + xy — x, find the second derivative 
matrices A; and Ao: 

0°F /dx* 0?F /dxdy 

0°F /dydx 07°F /dy* |~ 


A, 1s positive definite, so F, is concave up (= convex). Find the minimum point of 
F, and the saddle point of Fp (look where first derivatives are zero). 


Aes 


21. The graph of z = x* + y” is a bowl opening upward. The graph of z= x? —y’ is a 
saddle. The graph of z = —x* — y” is a bowl opening downward. What is a test on 
F (x,y) to have a saddle at (0,0)? 


22. Which values of c give a bowl and which give a saddle point for the graph of z = 
4x? + 12xy + cy”? Describe this graph at the borderline value of c. 


6.2 Tests for Positive Definiteness 


Which symmetric matrices have the property that x'Ax > 0 for all nonzero vectors x? 
There are four or five different ways to answer this question, and we hope to find all of 
them. The previous section began with some hints about the signs of eigenvalues. but 
that gave place to the tests on a, b, c: 


b 
b= f | is positive definite when a>0O and ac—b’?>0. 
Cc 


From those conditions, both eigenvalues are positive. Their product A A> is determinant 
ac — b* > 0, so the eigenvalues are either both positive or both negative. They must be 
positive because their sum is the trace a+c > 0. 

Looking at a and ac — b’, it is even possible to spot the appearance of the pivots. They 
turned up when we decomposed x! Ax into a sum of squares: 


b\*? ac—b? 
Sum of squares ax* + 2bxy+cy’ =a (x + °») oe a (1) 


Those coefficients a and (ac — b*)/a are the pivots for a 2 by 2 matrix. For larger 
matrices the pivots still give a simple test for positive definiteness: x! Ax stays positive 
when n independent squares are multiplied by positive pivots. 
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One more preliminary remark. The two parts of this hook were linked by the chapter 
on determinants. Therefore we ask what part determinants play. It is not enough to 
require that the determinant of A is positive. If a= c = —1 and b = 0. then detA = 1 
but A = —/ = negative definite. The determinant test is applied not only to A itself, 
giving ac — b* > 0, but also to the 1 by 1 submatrix a in the upper left-hand corner. 

The natural generalization will involve all n of the upper left submatrices of A: 


ot ai, a2 a43 
1 a2 
Aj = lan| ; A2= ; A3= |d21 22 a23|,°"', An =A. 


a21 422 


431 432 433 
Here is the main theorem on positive definiteness, and a reasonably detailed proof: 


6B Each of the following tests is a necessary and sufficient condition for the 
real symmetric matrix A to be positive definite: 


(I) x'kx > 0 for all nonzero real vectors x. 
(II) All the eigenvalues of A satisfy A; > 0. 
(III) All the upper left submatrices A; have positive determinants. 


(IV) All the pivots (without row exchanges) satisfy d, > 0. 


Proof. Condition I defines a positive definite matrix. Our first step shows that each 
eigenvalue will be positive: 


If Ax=Ax, then x'Ax=x'Ax=A||x'|l. 


A positive definite matrix has positive eigenvalues, since x'Ax > 0. 

Now we go in the other direction. If all A; > 0, we have to prove x'Ax > 0 for 
every vector x (not just the eigenvectors). Since symmetric matrices have a full set of 
orthonormal eigenvectors, any x is a combination cjx; +--+ +¢pX,. Then 


Ax = CjAxX, +++: + CyAXn = C1A4X1 ++ + Op Ankn- 
7 


Because of the orthogonality x!x; = 0, and the normalization x!x; = 1, 
g y X; i 


x'Ax = (cixt free + Git) (cyAyxy +--+ + CpAnXn) 


2 
eee io 


If every A; > 0, then equation (2) shows that x'Ax > 0. Thus condition II implies condi- 
tion I. 

If condition I holds, so does condition IIIT: The determinant of A is the product of 
the eigenvalues. And if condition I holds, we already know that these eigenvalues are 
positive. But we also have to deal with every upper left submatrix A;. The trick is to 
look at all nonzero vectors whose last n — k components are zero: 


A 
me | = xP Aung > 0. 


* * 


T T 
A =| 0| 
X X Xi 0 
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Thus A, is positive definite. Its eigenvalues (not the same /;!) must be positive. Its 
determinant is their product, so all upper left determinants are positive. 

If condition III holds, so does condition IV: According to Section 4.4, the kth pivot 
d; is the ratio of detA; to detA;,_. If the determinants are all positive, so are the pivots. 

If condition IV holds, so does condition I: We are given positive pivots, and must 
deduce that x'Ax > 0. This is what we did in the 2 by 2 case, by completing the square. 
The pivots were the numbers outside the squares. To see how that happens for symmetric 
matrices of any size, we go back to elimination on a symmetric matrix: A= LDL". 


Example 1. Positive pivots 2, 3, and 7 


2 -1 0 LL? 20, 101),12 1 -; 0 
A=|-1 2 -1}=|-} 1 0 3 0 1 —§%| =ZDL" 
2 4 
0 -1l 2 Oe =a. 4 3/|0 O 1 
I want to split x'Ax into x'LDL' x: 
Uu 1 —5 0 u u—5V 
If x= ]v],_ then L'x=|0 1 3 vi = v— aw 
w 0 0 1 w w 


So x'Ax is a sum of squares with the pivots 2, 3, and : as coefficients: 


1 \? 2.\ 4 
x'Ax = (L'™x)'D(L'x) =2 (« — a) + ; (» —- =) + =(w)’. 


Those positive pivots in D multiply perfect squares to make x!Ax positive. Thus condi- 
tion IV implies condition I, and the proof is complete. 


It is beautiful that elimination and completing the square are actually the same. Elim- 
ination removes x; from all later equations. Similarly, the first square accounts for all 
terms in x! Ax involving x,. The sum of squares has the pivots outside. The multipliers 
¢;; are inside! You can see the numbers = 5 and ae inside the squares in the example. 

Every diagonal entry a;; must be positive. As we know from the examples, however, 
itis far from sufficient to look only at the diagonal entries. 

The pivots dj; are not to be confused with the eigenvalues. For a typical positive 
definite matrix, they are two completely different sets of positive numbers, In our 3 by 3 
example, probably the determinant test is the easiest: 


Determinant test detA; =2, detAr=3, detA3=detA =4. 


The pivots are the ratios dj = 2, dz = 3, d= ;. Ordinarily the eigenvalue test is the 
longest computation. For this A we know the (’s are all positive: 


Eigenvalue test Ay =2-V2, h=2, A=24+ v2. 
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Even though it is the hardest to apply to a single matrix, eigenvalues can be the most 
useful test for theoretical purposes. Each test is enough by itself. 


Positive Definite Matrices and Least Squares 


I hope you will allow one more test for positive definiteness. It is already close. We 
connected positive definite matrices to pivots (Chapter 1), determinants (Chapter 4), and 
eigenvalues (Chapter 5). Now we see them in the least-squares problems in Chapter 3, 
coming from the rectangular matrices of Chapter 2. 

The rectangular matrix will be R and the least-squares problem will be Rx = b. It has 
m equations with m > n (square systems are included). The least-square choice x is the 
solution of R'R¥ = R'b. That matrix AR'R is not only symmetric but positive definite, 
as we now show—provided that the n columns of R are linearly independent: 


6C The symmetric matrix A is positive definite if and only if 
(V) There is a matrix R with independent columns such that A = R'R. 


The key is to recognize x'Ax as x'R'™Rx = (Rx)'(Rx). This squared length ||Rx||? is 
positive (unless x = 0), because R has independent columns. (If x is nonzero then Rx is 
nonzero.) Thus x'R'Rx > 0 and R'R is positive definite. 

It remains to find an R For which A = R'R. We have almost done this twice already: 


Elimination =A=LDL'=(LVD)(VDL'). Sotake R= VDL". 
This Cholesky decomposition has the pivots split evenly between L and L!. 
Eigenvalues A= QAQ'=(QVA)(VAQ'). SotakeR=VAQ'’. (3) 


A third possibility is R = QV AQ", the symmetric positive definite square root of A. 
There are many other choices, square or rectangular, and we can see why. If you multiply 
any R by a matrix Q with orthonormal columns, then (QR)'(QR) = R'Q'OR = R'IR = 
A. Therefore QR is another choice. 

Applications of positive definite matrices are developed in my earlier book Jntro- 
duction to Applied Mathematics and also the new Applied Mathematics and Scientific 
Computing (see www.wellesleycambridge.com). We mention that Ax = AMx 
arises constantly in engineering analysis. If A and WM are positive definite, this general- 
ized problem is parallel to the familiar Ax = Ax, and A > 0. M is a mass matrix for the 
finite element method in Section 6.4. 


Semidefinite Matrices 


The tests for semidefiniteness will relax x'Ax > 0, A > 0, d > 0, and det > 0, to allow 
zeros to appear. The main point is to see the analogies with the positive definite case. 
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6D_ Each of the following tests is a necessary and sufficient condition for a 
symmetric matrix A to be positive semidefinite: 
(I’) x'Ax > 0 for all vectors x (this defines positive semidefinite). 
(II’) All the eigenvalues of A satisfy A; > 0. 
(III’) No principal submatrices have negative determinants. 
(IV’) No pivots are negative. 
(V’) There is a matrix R, possibly with dependent columns, such that A = R'R. 
The diagonalization A = QAQ! leads to x'Ax = x'QAQ'x = y' Ay. If A has rank r, there 
are r nonzero A’s and r perfect squares in y'Ay = Ay} tet Apy?. 


Note. The novelty is that condition III’ applies to all the principal submatrices, not only 
those in the upper left-hand corner. Otherwise, we could not distinguish between two 
matrices whose upper left determinants were all zero: 


0 0 0 0 
f is positive semidefinite, and E | is negative semidefinite. 


A row exchange comes with the same column exchange to maintain symmetry. 
Example 2. 
2 -1 -1 
A=j]-1 2 -l is positive semidefinite, by all five tests: 


=l1 =I 2 


I’) xTAx = (x1 — x2)? + (x1 — x3)? + (x2 — 23)? > 0 (zero if xy = x2 = x3). 


(II’) The eigenvalues are A, = 0, Az = A3 = 3 (a zero eigenvalue). 


(III’) detA = 0 and smaller determinants are positive. 


2 -1 -1l 2 Oy 0 2 0 0 
(v’)A=|]-1 2 -1} > |0 . = — |0 3 O| (missing pivot). 
af 52 29 O.=), = 00 0 
(V’) A= R'R with dependent columns in R: 
2 -l1 -l 1 -1 0O 1 O -l 
—-1 2 -1l/=/]0 1 -1t}}-1 1 =O (1,1, 1) in the nullspace. 
—-1 -l 2 -1 0 1 0 -1 1 


Remark. The conditions for semidefiniteness could also be deduced from the origin con- 
ditions I-V for definiteness by the following trick: Add a small multiple of the identity 
giving a positive definite matrix A+ €/. Then let € approach zero. Since the determi- 
nants and eigenvalues depend continuously on €, they will be positive until the very last 
moment. At € = 0 they must still be nonnegative. 
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My class often asks about unsymmetric positive definite matrices. I never use that 
term. One reasonable definition is that the symmetric part 5(A +A") should be positive 
definite. That guarantees that the real parts of the eigenvalues are positive. But it is not 
necessary: A = [}{] has A > 0 but 3(A +A") = [} 7] is indefinite. 


If Ax = Ax, then xHAx = AxEx and xHAMx = AxHx. 


Adding, 5x4(A +A¥)x = (ReA)xtx > 0, so that Red > 0. 


Ellipsoids in n Dimensions 


Throughout this book, geometry has helped the matrix algebra. A linear equation pro- 
duced a plane. The system Ax = b gives an intersection of planes. Least squares gives 
a perpendicular projection. The determinant is the volume of a box. Now, for a positive 
definite matrix and its x'Ax, we finally get a figure that is curved. It is an ellipse in two 
dimensions, and an ellipsoid in n dimensions. 

The equation to consider is x!Ax = 1. If A is the identity matrix, this simplifies to 
x +x3 +++++x? = 1. This is the equation of the “unit sphere” in R”. If A = 4/, the 
sphere gets smaller. The equation changes to 4xt +-+.+4x? = 1. Instead of (1,0,...,0), 
it goes through (5,0, ...,0). The center is at the origin, because if x satisfies x'Ax = 1, 
so does the opposite vector —x. The important step is to go from the identity matrix to a 
diagonal matrix: 


Ellipsoid ForA=| 1 |, the equation is x'Ax = 4x7 +2x3+ $33 = L. 
1 
9 
Since the entries are unequal (and positive!) the sphere changes to an ellipsoid. 

One solution is x = (5,0,0) along the first axis. Another is x = (0,1,0). The major 
axis has the farthest point x = (0,0,3). It is like a football or a rugby ball, but not quite— 
those are closer to rT +x5 + 5x4 = ]. The two equal coefficients make them circular in 
the x,-x2 plane, and much easier to throw! 

Now comes the final step, to allow nonzeros away from the diagonal of A. 


Example 3. A = [3 4] and x'Ax = Su? + 8uv+ 5v? = 1. That ellipse is centered at 
u =v = 0, but the axes are not so clear. The off-diagonal 4s leave the matrix positive 
definite, but they rotate the ellipse—its axes no longer line up with the coordinate axes 
(Figure 6.2). We will show that the axes of the ellipse point toward the eigenvector of 
A. Because A = A!, those eigenvectors and axes are orthogonal. The major axis of the 
ellipse corresponds to the smallest eigenvalue of A. 

To locate the ellipse we compute A; = 1 and Az = 9. The unit eigenvectors are 
(1,—1)/V2 and (1,1)/V2. Those are at 45° angles with the u-v axes, and they are 
lined up with the axes of the ellipse. The way to see the ellipse properly is to rewrite 
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Figure 6.2: The ellipse x'Ax = Su* + 8uv +5v? = | and its principal axes. 


x'Ax =1: 


2 2 
u v u v 
New squares su + Buv4 v= (-—) +9( +) =l1. @€4 
: v2 yz) \n 2 = 
A = 1 and A = 9 are outside the squares. The eigenvectors are inside. This is different 
from completing the square to 5(u+ zy)? + ZV", with the pivots outside. 
The first square equals 1 at (1/\/2,—1/,/2) at the end of the major axis. The minor 
axis is one-third as long, since we need (3) to cancel the 9. 


Any ellipsoid x'Ax = 1 can be simplified in the same way. The key step is to diago- 
nalize A = QAQ'. We straightened the picture by rotating the axes. Algebraically, the 
change to y = Q' x produces a sum of squares: 


xTAx = (x'Q)A(Q'x) = y Ay =Aryp +++ + Any = 1. (5) 


The major axis has y; = 1/,./A, along the eigenvector with the smallest eigenvalue. 

The other axes are along the other eigenvectors. Their lengths are 1/\/Az,...,1/WAn- 
Notice that the A’s must be positive—the matrix must be positive definite—or these 
square roots are in trouble. An indefinite equation yt — Oy5 = | describes a hyperbola 
and not an ellipse. A hyperbola is a cross-section through a saddle, and an ellipse is a 
cross-section through a bowl. 

The change from x to y = Q'x rotates the axes of the space, to match the axes of 
the ellipsoid. In the y variables we can see that it is an ellipsoid, because the equation 
becomes so manageable: 


6E Suppose A = QAQ! with A; > 0. Rotating y = O'x simplifies x'Ax = 1: 
x'QAQ'x=1, ylAy=1, and Ajyi+---+Any2=1. 


This is the equation of an ellipsoid. Its axes have lengths 1/VA1,...,1/WAn 
from the center. In the original x-space they point along the eigenvectors of A. 
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The Law of Inertia 


For elimination and eigenvalues, matrices become simpler by elementary operations 
The essential thing is to know which properties of the matrix stay unchanged. When 
a multiple of one row is subtracted from another, the row space, nullspace. rant and 
determinant all remain the same. For eigenvalues, the basic operation was a similarity 
transformation A — S~!AS (or A > M~!AM). The eigenvalues are unchanged (and also 
the Jordan form). Now we ask the same question for symmetric matrices: What are the 
elementary operations and their invariants for x'Ax? 

The basic operation on a quadratic form is to change variables. A new vector y is 
related to x by some nonsingular matrix, x = Cy. The quadratic form becomes y'CTACy. 
This shows the fundamental operation on A: 


Congruence transformation A—CT'AC for some nonsingular C. (6) 


The symmetry of A is preserved, since C'AC remains symmetric. The real question is, 
What other properties are shared by A and C'AC? The answer is given by Sylvester’s 
law of inertia. 


6F C'AC has the same number of positive eigenvalues, negative eigenvalues, 
and zero eigenvalues as A. 


The signs of the eigenvalues (and not the eigenvalues themselves) are preserved by a 
congruence transformation. In the proof, we will suppose that A is nonsingular. Then 
C'AC is also nonsingular, and there are no zero eigenvalues to worry about. (Otherwise 
we can work with the nonsingular A + €/ and A — €/, and at the end let € — 0.) 


Proof. We want to borrow a trick from topology. Suppose C is linked to an orthogonal 
matrix Q by a continuous chain of nonsingular matrices C(t). Att =0 andt = 1, C(0) = 
C and C(1) =Q. Then the eigenvalues of C(t)'AC(r) will change gradually, as t goes 
from 0 to 1, from the eigenvalues of C'AC to the eigenvalues of Q'AQ. Because C(t) is 
never singular, none of these eigenvalues can touch zero (not to mention cross over it!). 
Therefore the number of eigenvalues to the right of zero, and the number to the left, is 
the same for C'AC as for Q'AQ. And A has exactly the same eigenvalues as the similar 
matrix Q~'AQ = O'AQ. 

One good choice for Q is to apply Gram-Schmidt to the columns of C. Then C = QR, 
and the chain of matrices is C(t) =tQ+(1—1t)QR. The family C(t) goes slowly through 
Gram-Schmidt, from QR to Q. It is invertible, because Q is invertible and the triangular 
factor t] + (1 —t)R has positive diagonal. That ends the proof. 


Example 4. Suppose A = J. Then C'AC = C'C is positive definite. Both J and C'C 
have n positive eigenvalues, confirming the law of inertia. 


Example 5. If A = [( °, |, then C™AC has a negative determinant: 
detC™AC = (detC™)(detA)(detC) = —(detC)? < 0. 
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Then C'AC must have one positive and one negative eigenvalue, like A. 
Example 6. This application is the important one: 


6G _ For any symmetric matrix A, the signs of the pivots agree with the signs 
of the eigenvalues. The eigenvalue matrix A and the pivot matrix D have the 
same number of positive entries, negative entries, and zero entries. 


We will assume that A allows the symmetric factorization A = LDL" (without row ex- 
changes). By the law of inertia, A has the same number of positive eigenvalues as D. 
But the eigenvalues of D are just its diagonal entries (the pivots). Thus the number of 
positive pivots matches the number of positive eigenvalues of A. 

That is both beautiful and practical. It is beautiful because it brings together (for 
symmetric matrices) two parts of this book that were previously separate: pivots and 
eigenvalues. It is also practical, because the pivots can locate the eigenvalues: 


A has positive pivots a eae 
itive pi 
eiiagens cae tease =:|3: 40 LD |3-8.- a 
A-—2I has a negative pivot 
0 7 8 0 7 6 


A has positive eigenvalues, by our test. But we know that Amin is smaller than 2, because 
subtracting 2 dropped it below zero. The next step looks at A — J, to see if Amin < 1. (It 
is, because A — J has a negative pivot.) That interval containing / is cut in half at every 
step by checking the signs of the pivots. 

This was almost the first practical method of computing eigenvalues. It was dominant 
about 1960, after one important improvement—to make A tridiagonal first. Then the 
pivots are computed in 27 steps instead of an. Elimination becomes fast, and the search 
for eigenvalues (by halving the intervals) becomes simple. The current favorite is the 
QR method in Chapter 7. 


The Generalized Eigenvalue Problem 


Physics, engineering, and statistics are usually kind enough to produce symmetric ma- 
trices in their eigenvalue problems. But sometimes Ax = 1x is replaced by Ax = 1Mx. 
There are two matrices rather than one. 

An example is the motion of two unequal masses in a line of springs: 


peas 0 

7 aaa 0 | a 2. eI 

dt or ge eatiaag u=0. (7) 
d?w 0 m| dt? |-1 2 

Ve eS 


When the masses were equal, m, = m2 = 1, this was the old system uv” + Au = 0. Now 
itis Mu” + Au = 0, with a mass matrix M. The eigenvalue problem arises when we look 
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for exponential solutions e'®’x: 


Mu"+Au=0 becomes M(i@)7e''x+Ael'x =0. (8) 


io and writing A for w”, this is an eigenvalue problem: 


0 
ai |= (9) 
0 my 
There is a solution when A — AM is singular. The special choice M = J brings back the 
usual det(A — AJ) = 0. We work out det(A — AM) with m; = 1 and mz = 2: 


3473 
5} ; 


Canceling e 


2 -l 
Generalized problem Ax = AMx | iS | — 


2-A -1 RED _ . = 
at? | —6A+3=0 gives A= 


For the eigenvector x (\/3 — 1,1), the two masses oscillate together—but the first mass 
only moves as far as V3 — 1 & .73. In the fastest mode, the components of x2 = (1+ 
/3, —1) have opposite signs and the masses move in opposite directions. This time the 
smaller mass goes much further. 

The underlying theory is easier to explain if M is split into R'R. (M is assumed to be 
positive definite.) Then the substitution y = Rx changes 


Ax =AMx=AR'Rx into AR-'y=AR'y. 


Writing C for R~!, and multiplying through by (R')~! = C', this becomes a standard 
eigenvalue problem for the single symmetric matrix C'AC: 


Equivalent problem CTACy = Ay. (10) 


The eigenvalues A; are the same as for the original Ax = AMx. and the eigenvectors are 
related by y; = Rx;. The properties of C'AC lead directly to thc properties of Ax = AMx, 
when A = A! and M is positive definite: 


1. The eigenvalues for Ax = AMx are real, because C'AC is symmetric. 
2. The A’s have the same signs as the eigenvalues of A, by the law of inertia. 


3. CTAC has orthogonal eigenvectors y;. So the eigenvectors of Ax = AMx have 
““M -orthogonality” x} Mx j= x} R'Rx j= yly j=. (11) 


A and M are being simultaneously diagonalized. If S has the x; in its columns, then 
SAS = A and S'MS =I. This is a congruence transformation, with S™ on the left, 
and not a similarity transformation with S~'. The main point is easy to summarize: As 
long as M is positive definite, the generalized eigenvalue problem Ax = —A Mx behaves 
exactly like Ax = Ax. 
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Problem Set 6.2 


Ik, 


10. 


11. 


For what range of numbers a and b are the matrices A and B positive definite? 


. Construct an indefinite matrix with its largest entries on the main diagonal: 


1 b —b 
A=|b 1 b with |b| < 1 can have detA < 0. 
—b b 1 


. Show from the eigenvalues that if A is positive definite, so is A* and so is A~!. 


. [fA and B are positive definite, then A+ B is positive definite. Pivots and eigenvalues 


are not convenient for A+ B. Much better to prove x'(A +B)x > 0. 


. From the pivots, eigenvalues, and eigenvectors of A = EB ap write A as R'R in three 


ways: (LVD)(VDL"), (@VA)(VWAQ"), and (QVAQ™)(QVAQ'). 


. IfA = QAQ' is symmetric positive definite, then R= QVAQ’ is its symmetric pos- 


itive definite square root. Why does R have positive eigenvalues? Compute R and 
verify R* = A for 


10 6 
6 10 


A= 


| and A= 


. If A is symmetric positive definite and C is nonsingular, prove that B = C'AC is also 


symmetric positive definite. 


. If A =R'R prove the generalized Schwarz inequality |x'Ay|* < (x!Ax)(yTAy). 


The ellipse u* + 4v? = 1 corresponds to A = ie ‘| . Write the eigenvalues and eigen- 
vectors, and sketch the ellipse. 


Reduce the equation 3u* — 2\/2uv + 2v? = 1 to a sum of squares by finding the 
eigenvalues of the corresponding A, and sketch the ellipse. 


12. 


13. 


14. 


15. 


16. 


17. 


18. 


19. 


20. 
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In three dimensions, Ajy7 + Aoy5 + Asy3 = 1 represents an ellipsoid when all A; > 0. 
Describe all the different kinds of surfaces that appear in the positive semidefinite 
case when one or more of the eigenvalues is zero. 


Write down the five conditions for a 3 by 3 matrix to be negative definite (—A is 
positive definite) with special attention to condition III: How is det(—A) related to 
detA? 


Decide whether the following matrices are positive definite, negative definite, semidef- 
inite, or indefinite: 


noe [C2 Oh 6 
i 6 => 0 4 
A=|254|, B= , C=-B, D=A 
aie Oo be = 
O-O 2: 3 


Is there a real solution to —x” — Sy” — 92? — 4xy — 6xz — 8yz = 1? 

Suppose A is symmetric positive definite and Q is an orthogonal matrix. True or 
false: 

(a) O'AQ isa diagonal matrix. 

(b) O'AQ is symmetric positive definite. 

(c) O'AQ has the same eigenvalues as A. 

(d) e~4 is symmetric positive definite. 


If A is positive definite and a; is increased, prove from cofactors that the determinant 
is increased. Show by example that this can fail if A is indefinite. 


From A = R'R. show for positive definite matrices that detA < a11a22---dny. (The 
length squared of column j of R is a;;. Use determinant = volume.) 


(Lyapunov test for stability of M) Suppose AM + M4A = —] with positive definite 
A. If Mx = Ax show that ReA < 0. (Hint: Multiply the first equation by x! and x.) 


Which 3 by 3 symmetric matrices A produce these functions f = x'Ax? Why is the 
first matrix positive definite but not the second one? 

(a) f= 2(x7 +5 4x3 — X1X2 — X2X3). 

(bh). = 2G +5 +x3 — X1X_ —X4X3—XX3). 

Compute the three upper left determinants to establish positive definiteness. Verify 


that their ratios give the second and third pivots. 


2.2. 0 
A= |2 5 3 
03 8 
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21. A positive definite matrix cannot have a zero (or even worse, a negative number) on 
its diagonal. Show that this matrix fails to have x'Ax > 0: 


41 1| |x 
Le x2 x3 1 0 2] |x is not positive when (x1,x%2,x3)=( , , ). 
25a es 


22. A diagonal entry aj; of a symmetric matrix cannot be smaller than all A’s. If it 
were, then A — a;;J would have eigenvalues and would be positive definite. But 
A—aj;jI hasa on the main diagonal. 


23. Give a quick reason why each of these statements is true: 
(a) Every positive definite matrix is invertible. 
(b) The only positive definite projection matrix is P = J. 
(c) A diagonal matrix with positive diagonal entries is positive definite. 


(d) A symmetric matrix with a positive determinant might not be positive definite! 


24. For which s and t do A and B have all A > 0 (and are therefore positive definite)? 


s —4 —4 t 3 0 
A= |-4 5s -4 and B= 3° 1°24 
-4 -4 5 0 4 t 


25. You may have seen the equation for an ellipse as (+)? + (7)? = 1. What are a and 
b when the equation is written as Mx? + Ary* = 1? The ellipse Ox? + 16y? = | has 
half-axes with lengths a = ,and b= 


26. Draw the tilted ellipse x7 + xy + y* = 1 and find the half-lengths of its axes from the 
eigenvalues of the corresponding A. 


27. With positive pivots in D, the factorization A = LDL’ becomes L\/D\VDL". (Square 
roots of the pivots give D = /D\/D.) Then C = Ly/’D yields the Cholesky factor- 
ization A = CC", which is “symmetrized LU”: 


4 8 


F C= 
rom 8 25 


i ; find A. From A= 


| find C. 


28. Inthe Cholesky factorization A = CC', with C = L/D, the square roots of the pivots 
are on the diagonal of C. Find C (lower triangular) for 


9 0 0 111 
A=]|0 1 2 and A=|1 2 2 
0 2 8 bP 2: 7. 


29. 


30. 


31. 


32. 


33. 


34. 


35. 


36. 
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The symmetric factorization A = LDL" means that x'Ax = x' LDL" x: 


[x y : ; A 7 [x y 7 / : mv 4 


0 1 
The left-hand side is ax” + 2bxy + cy”. The right-hand side is a(x + By)? ze y*. 
The second pivot completes the square! Test with a = 2,b = 4, c = 10. 


Xx 
: : 


Without multiplying A = [0088 —sin] [2 2] | cosé, siné | find 


sin@ cos@ —sin@ cos@ 


(a) the determinant of A. (b)_ the eigenvalues of A. 
(c) the eigenvectors of A. (d) areason why A is symmetric positive definite. 


For the semidefinite matrices 


2 -1 -l 111 
A= {-1l 2 —1| (rank 2) and B=1/1 1 1| (rank), 
—-1 -1 2 111 
write x'Ax as a sum of two squares and x! Bx as one square. 
Apply any three tests to each of the matrices 
i ae ae | De ik <2 
A=]}1 1 1 and B= 1 1 1], 
EQ 2.1.2 


to decide whether they are positive definite, positive semidefinite, or indefinite. 


For C = [3 a and A = E tl , confirm that C'AC has eigenvalues of the same signs 
as A. Construct a chain of nonsingular matrices C(t) linking C to an orthogonal 
Q. Why is it impossible to construct a nonsingular chain linking C to the identity 
matrix? 


If the pivots of a matrix are all greater than 1, are the eigenvalues all greater than 1? 
Test on the tridiagonal —1, 2, —1 matrices. 


Use the pivots of A — I to decide whether A has an eigenvalue smaller than 5: 


1 2 o> 10) 
A- a 3: OS OF 
O. “7 


An algebraic proof of the law of inertia starts with the orthonormal eigenvectors 
X1,...,Xp of A corresponding to eigenvalues A; > 0. and the orthonormal eigenvectors 
V1,-++5Vq OF C'AC corresponding to eigenvalues 1; < 0. 
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(a) To prove that the p + q vectors x|,...,Xp, Cy1,...,Cyg are independent, assume 
that some combination gives zero: 


AX, + +++ + apXp = biCyi +---+bgCyq (=z, say). 


Show that z'Az = Aaj feee+ Apa, > 0 and z!Az= Uy b+ +--- + [b? <0. 


(b) Deduce that the a’s and b’s are zero (proving linear independence). From that 
deduce p+q <n. 


(c) The same argument for the n — p negative A’s and the n — q positive L’s gives 
n—p+n—q <n. (We again assume no zero eigenvalues—which are handled 
separately). Show that p+q =n, so the number p of positive 2’s equals the 
number n — q of positive ’s—which is the law of inertia. 


37. If C is nonsingular, show that A and C'AC have the same rank. Thus they have the 
same number of zero eigenvalues. 


38. Find by experiment the number of positive, negative, and zero eigenvalues of 


I B 
B' 0 


when the block B (of order 5n) is nonsingular. 


A= 


39. Do A and C!AC always satisfy the law of inertia when C is not square? 


40. In equation (9) with m; = 1 and m2 = 2, verify that the normal modes are M- 
orthogonal: xIMxp =), 


41. Find the eigenvalues and eigenvectors of Ax = AMx: 
6 —3 pen A|4 1 
-3 6] 18]1 4 


42. If the symmetric matrices A and M are indefinite, Ax = AMx might not have real 
eigenvalues. Construct a 2 by 2 example. 


x. 


43. A group of nonsingular matrices includes AB and A~! if it includes A and B. “Prod- 
ucts and inverses stay in the group.” Which of these sets are groups? Positive definite 
symmetric matrices A, orthogonal matrices Q, all exponentials e'4 of a fixed matrix 
A, matrices P with positive eigenvalues, matrices D with determinant 1. Invent a 
group containing only positive definite matrices. 
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6.3 Singular Value Decomposition 


A great matrix factorization has been saved for the end of the basic course. ULV" joins 
with LU from elimination and QR from orthogonalization (Gauss and Gram-Schmidt). 
Nobody’s name is attached; A = UXV! is known as the “SVD” or the singular value 
decomposition. We want to describe it, to prove it, and to discuss its applications— 
which are many and growing. 

The SVD is closely associated with the eigenvalue-eigenvector factorization QAQ! of 
a positive definite matrix. The eigenvalues are in the diagonal matrix A. The eigenvector 
matrix Q is orthogonal (Q'Q = I) because eigenvectors of a symmetric matrix can be 
chosen to be orthonormal. For most matrices that is not true, and for rectangular matrices 
it is ridiculous (eigenvalues undefined). But now we allow the Q on the left and the on 
on the right to be any two orthogonal matrices U and V'—not necessarily transposes of 
each other. Then every matrix will split into A = UXV". 

The diagonal (but rectangular) matrix has eigenvalues from A'A, not from A! Those 
positive entries (also called sigma) will be o1,...,0,. They are the singular values of A. 
They fill the first r places on the main diagonal of X—when A has rank r. The rest of & 
is Zero. 

With rectangular matrices, the key is almost always to consider ATA and AA!. 


Singular Value Decomposition: Any m by n matrix A can be factored 
into 
A =UXV' = (orthogonal) (diagonal) (orthogonal). 


The columns of U (m by m) are eigenvectors of AA‘, and the columns of V (n 
by n) are eigenvectors of ATA. The r singular values on the diagonal of © (m 
by n) are the square roots of the nonzero eigenvalues of both AA! and ATA. 


Remark 1. For positive definite matrices, © is A and UXV! is identical to QAQ'. For 
other symmetric matrices, any negative eigenvalues in A become positive in ©. For 
complex matrices, © remains real but U and V become unitary (the complex version of 
orthogonal). We take complex conjugates in UHU = J and VEV = J and A = UXV#. 


Remark 2. U and V give orthonormal bases for all four fundamental subspaces: 


first r columns of U: column space of A 
last m—r _ columns of U: left nullspace of A 
first or columns of V: row space of A 
last n—r _ columns of V: nullspace of A 


Remark 3. The SVD chooses those bases in an extremely special way. They are more 
than just orthonormal. When A multiplies a column vj; of V, it produces 0; times a 
column of U. That comes directly from AV = UX, looked at a column at a time. 
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Remark 4, Eigenvectors of AA‘ and ATA must go into the columns of U and V: 
AA! = (UXV")(VE'UT) =Uxz'U! and, similarly, ATA=VETEZV!. (1) 


U must be the eigenvector matrix for AA’. The eigenvalue matrix in the middle is ££'— 
which is m by m with O;, ...,02 on the diagonal. 

From the ATA = VETEZV!, the V matrix must be the eigenvector matrix for A'A. The 
diagonal matrix £'E has the same 07,...,0,7, but it is n by n. 


Remark 5. Here is the reason that Av; = oju;. Start with ATAv; = =o; Vj: 
Multiply byA — AA™AV; = oF AV; (2) 


This says that Av; is an eigenvector of AAT! We just moved parentheses to (AA')(Av;). 
The length of this eigenvector Av; is 0;, because 


vIATAv; = ori F gives ||Av ||" = OF. 
So the unit eigenvector is Av;/o; = u;. In other words, AV = UX. 
Example 1. This A has only one column: rank r = 1. Then ¥ has only oj; = 3: 


1 2 
a. 13 


| WIN WIN 


Lol 


3 

0 1] = U3x323x1V 4. 
Z 0 
3 


A'A is 1 by 1, whereas AA! is 3 by 3. They both have eigenvalue 9 (whose square root 
is the 3 in X). The two zero eigenvalues of AA! leave some freedom for the eigenvectors 
in columns 2 and 3 of U. We kept that matrix orthogonal. 


2 -l 
Example 2. Now A has rank 2, and AA? = | fi: | with A = 3 and 1: 


-1l 1 0 
0 -1 1 


Notice 3 and 1. The columns of U are left singular vectors (unit eigenvectors of 
AA'). The columns of V are right singular vectors (unit eigenvectors of A‘A). 


I -=2) 1) /V6 
1 |-1 1] |/73 00 
n=l ule a A : [2s 


Application of the SVD 


We will pick a few important applications, after emphasizing one key point. The SVD is 
terrific for numerically stable computations. because U and V are orthogonal matrices. 
They never change the length of a vector. Since ||Ux||? =x 2, multiplication 
by U cannot destroy the scaling. 
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Of course © could multiply by a large o or (more commonly) divide by a small o, 
and overflow the computer. But still X is as good as possible. It reveals exactly what is 
large and what is small. The ratio Omax/Omin is the condition number of an invertible n 
by n matrix. The availability of that information is another reason for the popularity of 
the SVD. We come back to this in the second application. 


1. Image processing Suppose a satellite takes a picture, and wants to send it to Earth. 
The picture may contain 1000 by 1000 “pixels”—a million little squares, each with a 
definite color. We can code the colors, and send back 1,000,000 numbers. It is better to 
find the essential information inside the 1000 by 1000 matrix, and send only that. 

Suppose we know the SVD. The key is in the singular values (in X). Typically, some 
o’s are significant and others are extremely small. If we keep 20 and throw away 980, 
then we send only the corresponding 20 columns of U and V. The other 980 columns 
are multiplied in ULV" by the small o’s that are being ignored. We can do the matrix 
multiplication as columns times rows: 


A =UXV! =u o\vl +uopv} +--+» +u,0,v. (3) 


Any matrix is the sum of r matrices of rank 1. If only 20 terms are kept, we send 20 
times 2000 numbers instead of a million (25 to 1 compression). 

The pictures are really striking, as more and more singular values are included. At 
first you see nothing, and suddenly you recognize everything. The cost is in computing 
the SVD—this has become much more efficient, but it is expensive for a big matrix. 


2. The effective rank The rank of a matrix is the number of independent rows, and 
the number of independent columns. That can be hard to decide in computations! In 
exact arithmetic, counting the pivots is correct. Real arithmetic can be misleading—but 
discarding small pivots is not the answer. Consider the following: 


€ is small ae and a0 
I. 2 0 0 


The first has rank 1, although roundoff error will probably produce a second pivot. Both 
pivots will be small; how many do we ignore? The second has one small pivot, but we 
cannot pretend that its row is insignificant. The third has two pivots and its rank is 2, but 
its “effective rank” ought to be 1. 

We go to a more stable measure of rank. The first step is to use ATA or AA', which 
are symmetric but share the same rank as A. Their eigenvalues—the singular values 
squared—are not misleading. Based on the accuracy of the data, we decide on a toler- 
ance like 10~° and count the singular values above it—that is the effective rank. The 
examples above have effective rank 1 (when € is very small). 


€ 1 
é I1+€ 


3. Polar decomposition Every nonzero complex number z is a positive number r times 
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a number e’® on the unit circle: z = re'®. That expresses z in “polar coordinates.” If 
we think of z as a 1 by 1 matrix, r corresponds to a positive definite matrix and e!® 
corresponds to an orthogonal matrix. More exactly, since e’? is complex and satisfies 
e~!9 09 — 1. it forms a 1 by 1 unitary matrix. UU = I. We take the complex conjugate 
as well as the transpose, for U4. 

The SVD extends this “polar factorization” to matrices of any size: 


Every real square matrix can be factored into A = QS, where Q is orthogonal 
and S is symmetric positive semidefinite. If A is invertible then S is positive 
definite. 


For proof we just insert V'V = / into the middle of the SVD: 
A=UxV! =(UV')\(VxVv"). (4) 


The factor S=VXV! is symmetric and semidefinite (because ¥ is). The factor Q = U vi 
is an orthogonal matrix (because o'Q=vu'UV' =D). Inthe complex case, S becomes 
Hermitian instead of symmetric and Q becomes unitary instead of orthogonal. In the 
invertible case & is definite and so is S. 


Example 3. Polar decomposition: 


A=OS aa” be P| 
3 -1 1 OO} |-1 2 
Example 4. Reverse polar decomposition: 
A=s'O 1 —2 = 2 1) |0 -1 . 
3 -1 1 3) }1 O 


The exercises show how, in the reverse order. S changes but Q remains the same. Both 
S and S’ are symmetric positive definite because this A is invertible. 


Application of A = QS: A major use of the polar decomposition is in continuum 
mechanics (and recently in robotics). In any deformation, it is important to separate 
stretching from rotation, and that is exactly what QS achieves. The orthogonal matrix 
Q is a rotation, and possibly a reflection. The material feels no strain. The symmetric 
matrix S has eigenvalues 0;,...,0;, which are the stretching factors (or compression 
factors). The diagonalization that displays those eigenvalues is the natural choice of 
axes—called principal axes: as in the ellipses of Section 6.2. It is S that requires work 
on the material, and stores up elastic energy. 

We note that S? is A'A, which is symmetric positive definite when A is invertible. S 
is the symmetric positive definite square root of A'A, and Q is AS~!. In fact, A could be 
rectangular, as long as A'A is positive definite. (That is the condition we keep meeting, 
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that A must have independent columns.) In the reverse order A = S’Q, the matrix S’ is 
the symmetric positive definite square root of AA!. 


4. Least Squares For a rectangular system Ax = b. the least-squares solution comes 
from the normal equations A'AX = A'b. If A has dependent columns then A'A is not 
invertible and x is not determined. Any vector in the nullspace could be added to x. We 
can now complete Chapter 3, by choosing a “best” (shortest) x for every Ax = b. 

Ax = b has two possible difficulties: Dependent rows or dependent columns. With 
dependent rows, Ax = b may have no solution. That happens when b is outside the 
column space of A. Instead of Ax = b. we solve A'AX¥ = A'b. But if A has dependent 
columns, this * will not be unique. We have to choose a particular solution of ATAX = 
A'b, and we choose the shortest. 


The optimal solution of Ax = b is the minimum length solution of A'AX = A‘b. 


That minimum length solution will be called x*. It is our preferred choice as the best 
solution to Ax = b (which had no solution), and also to A'AX = Ab (which had too 
many). We start with a diagonal example. 


Example 5. A is diagonal, with dependent rows and dependent columns: 


o 0 0 0] |*! b 

wo ‘ X 
At=p is 10 o 0 0] |7l=|m 
0 00013 0 

X4 


The columns all end with zero. In the column space, the closest vector to b = (b,,b2,b3) 
is p = (bj,b2,0). The best we can do with Ax = b is to solve the first two equations, 
since the third equation is 0 = b3. That error cannot be reduced, but the errors in the first 
two equations will be zero. Then 


X1 = b/0; and x2 = b2/ 02. 


Now we face the second difficulty. To make x as short as possible, we choose the 
totally arbitrary x3 and x4 to be zero. The minimum length solution is x*: 


b,/0; 1/o, 0 0 


A* is pseudoinverse , | b2/o 0 1/o, 0 
. = = bo| . (5) 
x’ = Atb is shortest 0 0 0 O . 
0 OQ: <Q 2G) 


This equation finds x, and it also displays the matrix that produces x* from b. That 
matrix is the pseudoinverse A* of our diagonal A. Based on this example, we know L* 
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and x* for any diagonal matrix ©: 
O| 1/o; b;/o; 


B=] BS ” a 
O; 1/0, b,/ Or 


The matrix 2 is m by n, with r nonzero entries 0;. Its pseudoinverse X* is n by m, with 
r nonzero entries 1/o;. All the blank spaces are zeros. Notice that (£*)* is © again. 
That is like (A~!)~! =A, but here A is not invertible. 

Now we find x* in the general case. We claim that the shortest solution x* is always 
in the row space of A. Remember that any vector x can be split into a row space compo- 
nent x, and a nullspace component: x = x; -+X,. There are three important points about 
that splitting: 


1. The row space component also solves A'Ax, = A'b, because Ax, = 0. 
2. The components are orthogonal, and they obey Pythagoras’s law: 


|||]? = ||x,||? + llan||?, so Xis shortest when x, = 0. 


3. All solutions of ATA¥ = A'b have the same x,. That vector is x*. 


The fundamental theorem of linear algebra was in Figure 3.4. Every p in the column 
space comes from one and only one vector x; in the row space. All we are doing is to 
choose that vector, x* = x;,, as the best solution to Ax = b. 

The pseudoinverse in Figure 6.3 starts with b and comes back to x*. It inverts A where 
A is invertible—between row space and column space. The pseudoinverse knocks out 
the left nullspace by sending it to zero, and it knocks out the nullspace by choosing x, as 
Mie 

We have not yet shown that there is a matrix At that always gives x*—but there is. 
It will be n by m, because it takes b and p in R” back to x* in R". We look at one more 


example before finding At in general. 


Example 6. Ax = b is —x; + 2x2 + 2x3 = 18, with a whole plane of solutions. 

According to our theory, the shortest solution should be in the row space of A = 
[—1 2 2]. The multiple of that row that satisfies the equation is x* = (—2,4,4). There 
are longer solutions like (—2,5,3), (—2,7,1), or (—6,3,3), but they all have nonzero 
components from the nullspace. The matrix that produces x* from b = [18] is the pseu- 
doinverse A*. Whereas A was | by 3, this AX is 3 by 1: 


= 
and = A* [18] = | 4 |. (6) 
4 


+ 
At=[-1 2 2| = 


Ne} Xe)I) 
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\ 


\ 


Row space Column space\, 


A= 


op = Art 


8 © 


Nullspace of AT 


Figure 6.3: The pseudoinverse A* inverts A where it can on the column space. 


The row space of A is the column space of A*. Here is a formula for A*: 
If A = UXV' (the SVD), then its pseudoinverse is A* = VItU!. (7) 


Example 6 had o = 3—the square root of the eigenvalue of AA‘ = [9]. Here it is again 
with © and XT: 


aL 2 2 
a ane 
A= |—1 2 2| Lysyr = | 3 0 0| 2 1 2 
2 Be. so, 
3 3 3 
_l 2 2 a _l 
+77T yr 1 3 : 2 + 
Vxu'U a =a 0 1] =| 5|=A4 
Bo iu NG 2 
3 3 3 9 
The minimum length least-squares solution is xt = Atb =V=X*U'D. 


Proof. Multiplication by the orthogonal matrix U™ leaves lengths unchanged: 
\|Ax — b|| = ||UZVTx— bl] = ||ZVTx— UTI]. 


Introduce the new unknown y = V'x = V~!x, which has the same length as x. Then, 
minimizing ||Ax — b]| is the same as minimizing ||Xy — U'b]||. Now is diagonal and we 
know the best y*. It is yY = X+U"D so the best xt is Vy": 


Shortest solution = xt =Vyt =VxtU!D=A‘Db. 


Vy* is in the row space, and A'Axt = A'b from the SVD. 


374 Chapter 6 Positive Definite Matrices 


Problem Set 6.3 


Problems 1-2 compute the SVD of a square singular matrix A. 


1. Compute A‘A and its eigenvalues orm 0 and unit eigenvectors v1, v2: 


2. (a) Compute AA! and its eigenvalues O;, 0 and unit eigenvectors 1, u2. 


(b) Choose signs so that Av; = 0, and verify the SVD: 


(c) Which four vectors give orthonormal bases for C(A), N(A), C(A!), N(A‘)? 
Problems 3-5 ask for the SVD of matrices of rank 2. 


3. Find the SVD from the eigenvectors v1, v2 of A'A and Ay; = onn;: 


Fibonacci matrix A= 


4. Use the SVD part of the MATLAB demo eigshow (or Java on the course page 
web.mit.edu/18. 06) to find the same vectors v; and v2 graphically. 


5. Compute A!A and AA!, and their eigenvalues and unit eigenvectors, for 


1 1 

A= : 
01 1 

Multiply the three matrices ULV! to recover A. 


Problems 6-13 bring out the underlying ideas of the SVD. 


6. Suppose u1,...,u, and vj,...,v, are orthonormal bases for R”. Construct the matrix 
A that transforms each v; into u; to give Avy = u1,...,AV_n = Un. 


7. Construct the matrix with rank 1 that has Av = 12u for v = 5(1, 1,1,1) and u= 
$(2, 2,1). Its only singular value is 0; = 


8. Find UXV! if A has orthogonal columns w,...,W, of lengths 01,..., On. 
9. Explain how UXV! expresses A as a sum of r rank-1 matrices in equation (3): 


T T 
A= O,U,V; +:::+0,UrV,. 


10. 


11. 


12. 


13. 
14. 
15. 


16. 
17. 


18. 


19. 
20. 


6.3 Singular Value Decomposition 375 


Suppose A is a 2 by 2 symmetric matrix with unit eigenvectors uw; and u2. If its 
eigenvalues are A; = 3 and A, = —2, what are U, £, and V!? 


Suppose A is invertible (with 0; > 02 > 0). Change A by as small a matrix as possible 
to produce a singular matrix Ag. Hint: U and V do not change: 


; Oo! = 
Find Ao from A= Ei uw és E va] : 
2 


(a) If A changes to 4A, what is the change in the SVD? 
(b) What is the SVD for A! and for A~!? 


Why doesn’t the SVD for A +/ just use © + J? 


Find the SVD and the pseudoinverse 0* of the m by n zero matrix. 


11 
0 0]. 


Find the SVD and the pseudoinverse VZtU! of 


A=|I 11 1], Belo Ne aad = 
100 


If an m by n matrix Q has orthonormal columns, what is QT? 


Diagonalize A‘A to find its positive definite square root S = VE!/2V7 and its polar 
decomposition A = QS: 


— 1 110 6 
9/1010. SN 


What is the minimum-length least-squares solution xt = A*} to the following? 


10 OJ JC 0 
Ax=|1 0 O| |D| = [2 
DD) ee 2 


You can compute A‘, or find the general solution to A'AX¥ = A‘ and choose the 
solution that is in the row space of A. This problem fits the best plane C+ Dt + Ez to 
b=Oandalsob=2 att=z=O(andb=2 att=z=1). 


(a) If A has independent columns, its left-inverse (A'A)~'A? is AT. 

(b) If A has independent rows, its right-inverse A'(AA')~! is A*. 

In both cases, verify that x* = Atb is in the row space. and A'Axt = Alb. 
Split A = UXV! into its reverse polar decomposition QS’. 


Is (AB)* = BTA™ always true for pseudoinverses? I believe not. 
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21. Removing zero rows of U leaves A = LU, where the r columns or L span the column 
space of A and the r rows of U span the row space. Then A™ has the explicit formula 
OREO Gas wate @ Ses Oa 
Why is A*d in the row space with U! at the front? Why does A'AA*b = A'D, so 
that x* = A*D satisfies the normal equation as it should? 


22. Explain why AA* and ATA are projection matrices (and therefore symmetric). What 
fundamental subspaces do they project onto? 


6.4 Minimum Principles 


In this section we escape for the first time from linear equations. The unknown x will not 
be given as the solution to Ax = b or Ax = Ax. Instead, the vector x will be determined 
by a minimum principle. 

It is astonishing how many natural laws can be expressed as minimum principles. Just 
the fact that heavy liquids sink to the bottom is a consequence of minimizing their po- 
tential energy. And when you sit on a chair or lie on a bed, the springs adjust themselves 
so that the energy is minimized. A straw in a glass of water looks bent because light 
reaches your eye as quickly as possible. Certainly there are more highbrow examples: 
The fundamental principle of structural engineering is the minimization of total energy.! 

We have to say immediately that these “energies” are nothing but positive definite 
quadratic functions. And the derivative of a quadratic is linear. We get back to the 
familiar linear equations, when we set the first derivatives to zero. Our first goal in 
this section is to find the minimum principle that is equivalent to Ax = b, and the 
minimization equivalent to Ax = Ax. We will be doing in finite dimensions exactly 
what the theory of optimization does in a continuous problem, where “first derivatives 
= 0” gives a differential equation. In every problem, we are free to solve the linear 
equation or minimize the quadratic. 

The first step is straightforward: We want to find the “parabola” P(x) whose minimum 
occurs when Ax = b. If A is just a scalar, that is easy to do: 


1 dP 
The graph of P(x) = sax —bx has zero slope when ae Ax—b=0. 
Xx 


This point x = A~'b will be a minimum if A is positive. Then the parabola P(x) opens 
upward (Figure 6.4). In more dimensions this parabola turns into a parabolic bowl (a 
paraboloid). To assure a minimum of P(x), not a maximum or a saddle point, A must be 
positive definite! 


'T am convinced that plants and people also develop in accordance with minimum principles. Perhaps civilization 
is based on a law of least action. There must be new laws (and minimum principles) to be found in the social 
sciences and life sciences. 
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6H _ If A is symmetric positive definite, then P(x) = 5xTAx —x"b reaches its 
minimum at the point where Ax = b. At that point Phin = — sbTA~ |b. 


P(x) = $Ax* — br P(x) = $27 Ax — 27 


v2 


Prin = —50°/A wy Prin = ~ 3b" AD 


Figure 6.4: The graph of a positive quadratic P(x) is a parabolic bowl. 


Proof. Suppose Ax = b. For any vector y, we show that P(y) > P(x): 
P(y) — P(x) = syty —y'b— stan +x'b 
= syty —ylAx+ stan (set b = Ax) (1) 
=50-9"A0-2), 


This can’t be negative since A is positive definite—and it is zero only if y—x = 0. At all 
other points P(y) is larger than P(x), so the minimum occurs at x. 


Example 1. Minimize P(x) = ra — xX1x2 +x3 — byx; — box2. The usual approach, by 
calculus, is to set the partial derivatives to zero. This gives Ax = b: 


OP/Ox, = 2x; —x2—b; = 0 2 —Il) |x by 
means = . (2) 
OP/Oxz = —x; + 2x2 —b2 = 0 —] 2 x2 oD) 


Linear algebra recognizes this P(x) as 5xTAx —x'b, and knows immediately that Ax = b 
gives the minimum. Substitute x = A~'b into P(x): 


1 1 
Minimum value Prin = 5(A 'b)"A(A™'b) —(A-'b)"b = — bia ib. (3) 


In applications, 5xTAx is the internal energy and —x'b is the external work. The system 
automatically goes to x = A~'b, where the total energy P(x) is a minimum. 


Minimizing with Constraints 


Many applications add extra equations Cx = d on top of the minimization problem. 
These equations are constraints. We minimize P(x) subject to the extra requirement 
Cx = d. Usually x can’t satisfy n equations Ax = b and also @ extra constraints Cx = d. 
We have too many equations and we need ¢@ more unknowns. 
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Those new unknowns yj,...,y¢ are called Lagrange multipliers. They build the 
constraint into a function L(x, y). This was the brilliant insight of Lagrange: 


1 
L(x,y) = P(x) +y!(Cx—d) = 5x Ax —x'bh+x'cly—yla. 


That term in L is chosen exactly so that 0L/dy = 0 brings back Cx = d. When we set 
the derivatives of L to zero, we have n+ ¢ equations for n+ ¢ unknowns x and y: 


Constrained OL/ox=0: Ax+Cly=b 


A 
minimization OL/dy=0: Cx =d i 


The first equations involve the mysterious unknowns y. You might well ask what they 
represent. Those “dual unknowns” y tell how much the constrained minimum F¢/ min 
(which only allows x when Cx = d) exceeds the unconstrained Pin (allowing all x): 


ly 
Sensitivity of minimum Pej min = Pmin + 5) T(CA~!b —d) > Prin- (5) 


Example 2. Suppose P(x;,x2) = 
This unconstrained problem has n 
Ax = b just gives x; = 0 and x2 = 0. 

Now add one constraint c}x; + c2x2 = d. This puts x on a line in the x;-x2 plane. 
The old minimizer x; = x2 = 0 is not on the line. The Lagrangian function L(x, y) = 


5x4 + 5x5 + y(cx1 + c2x2 —d) hasn+¢ =2-+ 1 partial derivatives: 


5x + 5X5. Its smallest value is certainly Phin = 0. 
— A — 


2 
sual 
2s I, and b=0. So the minimizing equation 


OL/dx, =0 xj +c1y=0 
OL/Ox2 =0 x2 +c2y =0 (6) 
OL/dy =0 cyx; teox2 = d. 
Substituting x; = —c,y and x2 = —c2y into the third equation gives —ciy — chy =d. 
—d cid cod 
Solution = ~~ se x2 = : , 
“Bra "Sta @ ate © 
The constrained minimum of P = 5x1 x is reached at that solution point: 
6 Lyi Icfd? +d? 1 @ (8) 
= Xe = = 
= od ee (cf +c3)? 2ci+c5 


This equals — syd as predicted in equation (5), since b = 0 and Pin = 0. 


Figure 6.5 shows what problem the linear algebra has solved, if the constraint keeps 
x ona line 2x; — x2 =5. We are looking for the closest point to (0,0) on this line. The 
solution is x = (2,—1). We expect this shortest vector x to be perpendicular to the line, 
and we are right. 
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P= 3(aj + 23) 


Figure 6.5: Minimizing 5 lll for all x on the constraint line 2x) — x2 = 5. 


Least Squares Again 


In minimization, our big application is least squares. The best x is the vector that mini- 
mizes the squared error E* = ||Ax — b||. This is a quadratic and it fits our framework! I 
will highlight the parts that look new: 


Squared error E? = (Ax—b)'(Ax—b) =x! ATAx—2x'ATb+b'b. (9) 
Compare with 5xTAx —x"b at the start of the section, which led to Ax = b: 
[A changes to A‘A] [b changes to A'Db| [b'b is added| : 


The constant b'b raises the whole graph—this has no effect on the best X. The other 
two changes, A to A‘A and b to A'b, give a new way to reach the least-squares equation 
(normal equation). The minimizing equation Ax = b changes into the 


Least-squares equation A'Ax=A'b. (10) 


Optimization needs a whole book. We stop while it is pure linear algebra. 


The Rayleigh quotient 


Our second goal is to find a minimization problem equivalent to Ax = Ax. That is not so 
easy. The function to minimize cannot be a quadratic, or its derivative would be linear— 
and the eigenvalue problem is nonlinear (A times x). The trick that succeeds is to divide 
one quadratic by another one: 

xTAx 
T 


Rayleigh quotient Minimize R(x) = ; 
aes 

61 Rayleigh’s Principle: The minimum value of the Rayleigh quotient is 

the smallest eigenvalue A. R(x) reaches that minimum at the first eigenvector 

x1 of A: 


xtAx, xlAyx 
Minimum where Ax; = Ax, R(m) = Pees Ay. 
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If we keep x! Ax = 1, then R(x) is a minimum when x! x = ||x||? is as large as possible. 
We are looking for the point on the ellipsoid x'Ax = 1 farthest from the origin—the 
vector x of greatest length. From our earlier description of the ellipsoid, its longest axis 
points along the first eigenvector. So R(x) is a minimum at x. 

Algebraically, we can diagonalize the symmetric A by an orthogonal matrix: QTAQ = 
A. Then set x = Qy and the quotient becomes simple: 


(Qy)TA(Qy) _ yTAy _ Aryt tes + Any 
(QO) ty ey 


The minimum of R is A,, at the point where y; = 1 and yp = --- = y, = 0: 


R(x) = (11) 


At all points Ai(yt ty3 tess +92) < (Any + Any +++ + Any). 


The Rayleigh quotient in equation (11) is never below A, and never above A, (the largest 
eigenvalue). Its minimum is at the eigenvector x; and its maximum is at x,: 


xT Ax, x! Agi 


xy XX, 


Maximum where Ax, = AyXn R= —a we 

One small yet important point: The Rayleigh quotient equals a,;, when the trial vector 
is x = (1,0,...,0). So a;; (on the main diagonal) is between A, and A,. You can see this 
in Figure 6.6, where the horizontal distance to the ellipse (where a\1x* = 1) is between 
the shortest distance and the longest distance: 


1 __} 
Tie Jaa STi 


The diagonal entries of any symmetric matrix are between i, and i,. We drew Figure 
6.6 for a 2 by 2 positive definite matrix to see it clearly. 


whichis A, < ayy <A». 


Intertwining of the Eigenvalues 


The intermediate eigenvectors x2,...,X,—, are saddle points of the Rayleigh quotient 
(zero derivatives, but not minima or maxima). The difficulty with saddle points is that 
we have no idea whether R(x) is above or below them. That makes the intermediate 
eigenvalues Az,...,A,—1 harder to estimate. 

For this optional topic, the key is to find a constrained minimum or maximum. The 
constraints come from the basic property of symmetric matrices: x; is perpendicular to 
the other eigenvectors. 


6J The minimum of R(x) subject to x'x; = 0 is A2. The minimum of R(x) 
subject to any other constraint x'v = 0 is not above Ay: 


Az = min R(x) and An > min R(x). (12) 
x! x1=0 x! y=0 
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Figure 6.6: The farthest x =x, /\/A; and the closet x = x,//An both give x!Ax = x'Ax = 1. These are the major 
axes of the ellipse. 


This “maximin principle” makes A the maximum over all v of the minimum of R(x) with 
x!y =0. That offers a way to estimate A without knowing A). 


Example 3. Throw away the last row and column of any symmetric matrix: 


A, (A) =2-—v2 = 

(A) v2 aR ies 2 -1]A1(B)=1 
A,(A) =2 A=|-1 2 -—1] becomes B= -1 2 |a(p=3 
Aa(A) =2 40/2 0 -1 2 ane 


The second eigenvalue Aj(A) = 2 is above the lowest eigenvalue A,(B) = 1. The lowest 
eigenvalue A;(A) = 2— V2 is below A,(B). So A;(B) is caught between. 

This example chose v = (0,0, 1) so the constraint x'v = 0 knocked out the third com- 
ponent of x (thereby reducing A to B). 

The complete picture is an intertwining of eigenvalues: 


A\(A) < Ai(B) < Aa(A) < Ao(B) < +++ < An-1(B) < An(A). (13) 


This has a natural interpretation for an ellipsoid, when it is cut by a plane through the 
origin. The cross section is an ellipsoid of one lower dimension. The major axis Of this 
cross section cannot be longer than the major axis of the whole ellipsoid: A,(B) > 4; (A). 
But the major axis of the cross section is at least as long as the second axis of the original 
ellipsoid: A,(B) < A,(A). Similarly the minor axis of the cross section is smaller than 
the original second axis, and larger than the original minor axis: A2(A) < A2(B) < A3(A). 

You can see the same thing in mechanics. When springs and masses are oscillating, 
suppose one mass is held at equilibrium. Then the lowest frequency is increased but not 
above Az. The highest frequency is decreased, but not below A,_1. 

We close with three remarks, I hope your intuition says that they are correct. 
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Remark 1. The maximin principle extends to j-dimensional subspaces Sj: 


Maximum of minimum Aj+1 = max min 9] (14) 
all S; |xLS; 


Remark 2. There is also a minimax principle for A,._ ;: 


Minimum of maximum An—j = min max RCs) (15) 
all S; | x LS; 
If j = 1, we are maximizing R(x) over one constraint x'v = 0. That maximum is between 
the unconstrained A,,_; and A,,. The toughest constraint makes x perpendicular to the top 
eigenvector v = x,. Then the best x is the next eigenvector x,_1;. The “minimum of the 
maximum” is A,_1. 
Remark 3. For the generalized problem Ax = AMx, the same principles hold if M is 
positive definite. In the Rayleigh quotient, x'x becomes x! Mx: 
ub 


x Ax 
x'Mx 


Rayleigh quotient = Minimizing R(x) = gives A,(M~'A). (16) 


Even for unequal masses in an oscillating system (M ¥ J), holding one mass at equilib- 
rium will raise the lowest frequency and lower the highest frequency. 


Problem Set 6.4 


1. Consider the system Ax = b given by 


2 -l1 O xX] 
—1 2 -1] |x} = 
0 -l 2 x3 4 


Construct the corresponding quadratic P(x;,x2,x3), compute its partial derivatives 
OP/Ox;, and verify that they vanish exactly at the desired solution. 

2. Complete the square in P = 5xTAx b= 5 (x —A~'b)™A(x—A~'b) + constant. 
This constant equals Pin because the term before it is never negative. (Why?) 


3. Find the minimum, if there is one of P; = 5x? +xy+y? —3y and P) = 5x? —3y. What 


matrix A is associated with P,? 


4. (Review) Another quadratic that certainly has its minimum at Ax = b is 
1 1 1 
Q(x) = ) \|Ax — b||? = 5 A Ax x A b+ 5b. 


Comparing Q with P, and ignoring the constant 5b'b, what system of equations do 
we get at the minimum of Q? What are these equations called in the theory of least 
squares? 


10. 


11. 


12. 


13. 


14. 


15. 
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For any symmetric matrix A, compute the ratio R(x) for the special choice x = 
(1,...,1). How is the sum of all entries a;; related to A; and A,? 


With A = [ *, 5,'], find a choice of x that gives a smaller R(x) than the bound A; < 2 
that comes from the diagonal entries. What is the minimum value of R(x)? 


If B is positive definite, show from the Rayleigh quotient that the smallest eigenvalue 
of A+ B is larger than the smallest eigenvalue of A. 


If A, and LW; are the smallest eigenvalues of A and B, show that the smallest eigen- 
value 0, of A+ B is at least as large as A; + HW). (Try the corresponding eigenvector 
x in the Rayleigh quotients.) 


Note. Problems 7 and 8 are perhaps the most typical and most important results 
that come easily from Rayleigh’s principle, but only with great difficulty from the 
eigenvalue equations themselves. 


If B is positive definite, show from the minimax principle (12) that the second small- 
est eigenvalue is increased by adding B : A7(A+ B) > A,(A). 


If you throw away two rows and columns of A, what inequalities do you expect 
between the smallest eigenvalue u of the new matrix and the original 2’s? 


Find the minimum values of 


2 2 
X7 —XixX0 +x 
R(x) = 1 and SR) = 
X41 X5 


xy xIx2 + x5 
ane 
2x45 +5 


Prove from equation (11) that R(x) is never larger than the largest eigenvalue A. 


The minimax principle for A; involves j-dimensional subspaces Sj: 
Equivalent to equation (15) A; = min [max Rs) ; 
S; [xin S; 
(a) If A; is positive, infer that every S; contains a vector x with R(x) > 0. 
(b) Deduce that S; contains a vector y = C~!x with y'cTACy/yly > 0. 
(c) Conclude that the jth eigenvalue of C'AC, from its minimax principle, is also 


positive—proving again the law of inertia in Section 6.2. 


Show that the smallest eigenvalue A, of Ax = A Mx is not larger than the ratio a); /m11 
of the corner entries. 


Which particular subspace Sz in Problem 13 gives the minimum value Az? In other 
words, over which S2 is the maximum of R(x) equal to Az? 
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16. (Recommended) From the zero submatrix decide the signs of the n eigenvalues: 


0 - O01 
ai 0 2 

0 0 0 

1 2 n 


17. (Constrained minimum) Suppose the unconstrained minimum x = A~'b happens to 
satisfy the constraint Cx = d. Verify that equation (5) correctly gives Po/min = Pmin’ 
the correction term is zero. 


6.5 The Finite Element Method 


There were two main ideas in the preceding section on minimum principles: 
(i) Solving Ax = b is equivalent to minimizing P(x) = 5xTAx —x'b. 
(ii) Solving Ax = A,x is equivalent to minimizing R(x) = x'Ax/x'x. 


Now we try to explain how these ideas can be applied. 

The story is a long one, because these principles have been known for more than a 
century. In engineering problems like plate bending, or physics problems like the ground 
state (eigenfunction) of an atom, minimization was used to get a rough approximation 
to the true solution. The approximations had to be rough; the computers were human. 
The principles (i) and (ii) were there, but they could not be implemented. 

Obviously the computer was going to bring about a revolution. It was the method 
of finite differences that jumped ahead, because it is easy to “discretize” a differential 
equation. Already in Section 1.7, derivatives were replaced by differences. The physical 
region is covered by a mesh, and wu” = f(x) became uj41 —2uj+uj-1 =h’f;. The 
1950s brought new ways to solve systems Au = f that are very large and very sparse— 
algorithms and hardware are both much faster now. 

What we did not fully recognize was that even finite differences become incredibly 
complicated for real engineering problems, like the stresses on an airplane. The real 
difficulty is not to solve the equations, but to set them up. For an irregular region we 
piece the mesh together from triangles or quadrilaterals or tetrahedra. Then we need a 
systematic way to approximate the underlying physical laws. The computer has to help 
not only in the solution of Au = f and Ax = Ax, but in its formulation. 

You can guess what happened. The old methods came back, with a new idea and a 
new name. The new name is the finite element method. The new ides uses more of 
the power of the computer—in constructing a discrete approximation, solving it, and 
displaying the results—than any other technique in scientific computation’. If the basic 


Please forgive this enthusiasm: I know the method may not be immortal. 
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idea is simple, the applications can be complicated. For problems on this scale, the one 
undebatable point is their cost—I am afraid a billion dollars would be a conservative 
estimate of the expense so far. I hope some readers will be vigorous enough to master 
the finite element method and put it to good use. 


Trial Functions 


Starting from the classical Rayleigh-Ritz principle, | will introduce the new idea of finite 
elements. The equation can be —u = f(x) with boundary conditions u(0) = u(1) = 
0. This problem is infinite-dimensional (the vector b is replaced by a function f, and 
the matrix A becomes —d*/dx*). We can write down the energy whose minimum is 
required, replacing inner products v! f by integrals of v(x) f(x): 


1 
Total energy P(v) = —yif= 5 sf ve) \)dx — [» v(x) f(x)dx. C1) 


P(v) is to be minimized over all functions v(x) that satisfy v(0) = v(1) = 0. The function 
that gives the minimum will be the solution u(x). The differential equation has been 
converted to a minimum principle, and it only remains to integrate by parts: 


[vorrar= f oPax— bis SO Po)= fi Fo/@o)+ v0 700) es 


The term vv’ is zero at both limits, because v is. Now [(v’(x))?dx is positive like x™Ax. 
We are guaranteed a minimum. 

To compute the minimum exactly is equivalent to solving the differential equation ex- 
actly. The Rayleigh-Ritz principle produces an n-dimensional problem by choosing only 
n trial functions V,(x),...,Vn(x). From all combinations V = y,Vi (x) +--+ + ynVn(x), 
we look for the particular combination (call it U) that minimizes P(V). This is the key 
idea, to minimize over a subspace of V’s instead of over all possible v(x). The function 
that gives the minimum is U(x). We hope and expect that U(x) is near the correct u(x). 

Substituting V for v, the quadratic turns into 


dU a yV{(x) +++ tynVi(x)) "dx [ (rvite) + tanta) Fae (2) 


The trial functions V are chosen in advance. That is the key step! The unknowns 
Y1,---;¥n go into a vector y. Then P(V) = sy Ay —y'b is recognized as one of the 
quadratics we are accustomed to. The matrix entries A;; are i. ye Vidx = coefficient of 
yiy;. The components b; are [ V; fdx. We can certainly find the minimum of 5y™Ay—y™b 
by solving Ay = b. Therefore the Rayleigh-Ritz method has three steps: 


1. Choose the trial functions V,,...,Vp. 


2. Compute the coefficients Aj; and bj. 
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3. Solve Ay = b to find U(x) = y,Vi(x) +++: +yy_Vn(x). 


Everything depends on step 1. Unless the functions V;(x) are extremely simple, the 
other steps will be virtually impossible. And unless some combination of the V; is close 
to the true solution u(x), those steps will be useless. To combine both computability 
and accuracy, the key idea that makes finite elements successful is the use of piecewise 
polynomials as the trial functions V(x). 


Linear Finite Elements 


The simplest and most widely used finite element is piecewise linear. Place nodes at 
the interior points x) = h,x2 = 2h,...,xX, =nh, just as for finite differences. Then V; is 
the “hat function” that equals | at the node x;, and zero at all the other nodes (Figure 
6.7a). It is concentrated in a small interval around its node, and it is zero everywhere 
else (including x = 0 and x = 1). Any combination y,V; +---+ y,V, must have the value 
y; at node j (the other V’s are zero there), so its graph is easy to draw (Figure 6.7b). 


Va(z) Vie) =ywMit---+y5V5 


0 x4 = 4h 1 0 1 


Figure 6.7: Hat functions and their linear combinations. 


Step 2 computes the coefficients Aj; = { V; Vidx in the “stiffness matrix” A. The slope 
Vv; equals 1/h in the small interval to the left of x;, and —1/h in the interval to the right. 
If these “double intervals” do not overlap, the product V/ Vv; is zero and A;; = 0. Each 
hat function overlaps itself and only two neighbors: 


Diasonal’ 43 Ay = f ViVi = Vax f LY" ee? 
lagona I=] i= iV; ax = h Xx h pe 


1 —] —] 
Off-diagonal i= j+1 Ay= [ viviar= | (5) (=) dx=—-. 


Then the stiffness matrix is actually tridiagonal: 


Stiffness matrix A=- —-1 2 -1 


a es 
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This looks just like finite differences! It has led to a thousand discussions about the 
relation between these two methods. More complicated finite elements—polynomials of 
higher degree. defined on triangles or quadrilaterals for partial differential equations— 
also produce sparse matrices A. You could think of finite elements as a systematic way 
to construct accurate difference equations on irregular meshes. The essential thing is the 
simplicity of these piecewise polynomials. Inside every element, their slopes arc easy to 
find and to integrate. 

The components b; on the right side are new. Instead of just the value of f at x;, 
as for finite differences, they are now an average of f around that point: bj = | Vj fdx. 
Then, in step 3, we solve the tridiagonal system Ay = b, which gives the coefficients in 
the minimizing trial function U = yi Vj +--+: +ynVy. Connecting all these heights y; by 
a broken line, we have the approximate solution U(x). 

Example 1. w” = 2 with u(0) = u(1) = 0, and solution u(x) = x — x’. 

The approximation will use three intervals and two hat functions, with h = i. The 
matrix A is 2 by 2. The right side requires integration of the hat function times f(x) = 2. 
That produces twice the area 5 under the hat: 


2. Sail 
al. 2 


| and b= 


WIN WIN 


The solution to Ay = b is y= (G, 5). The best U(x) is sv + §V2, which equals § at the 


mesh points. This agrees with the exact solution u(x) = x —x* = ; — 5 


In a more complicated example, the approximation will not be exact at the nodes. 
But it is remarkably close. The underlying theory is explained in the author’s book An 
Analysis of the Finite Element Method (see www.wellesleycambridge.com) written 
jointly with George Fix. Other books give more detailed applications, and the subject 
of finite elements has become an important part of engineering education. It is treated 
in Introduction to Applied Mathematics, and also in my new book Applied Mathematics 
and Scientific Computing. There we discuss partial differential equations, where the 
method really comes into its own. 


Eigenvalue Problems 


The Rayleigh-Ritz idea—to minimize over a finite-dimensional family of V’s in place 
of all admissible v’s—is also useful for eigenvalue problems. The true minimum of the 
Rayleigh quotient is the fundamental frequency A;. Its approximate minimum A, will 
be larger—because the class of trial functions is restricted to the V’s. This step was 
completely natural and inevitable: to apply the new finite element ideas to this long- 
established variational form of the eigenvalue problem. 

The best example of an eigenvalue problem has u(x) = sinzx and A, = 27: 


Eigenfunction u(x) —u"=Au, with u(0)=u(1)=0. 
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That function sin zx minimizes the Rayleigh quotient v'Av/v'v: 


_ Jov@)(—v"@)dx _ fo’ @)Pax 
= I > oa 

Io (v(@))?dx Jo (v(x)? dx 
This is a ratio of potential to kinetic energy, and they are in balance at the eigenvector. 
Normally this eigenvector would be unknown, and to approximate it we admit only the 
trial candidates V = y,V, +---+ nV): 


Io (Vit---+y¥nVn)2dx  y'My 


Now we face a matrix problem: Minimize y'Ay/y'My. With M =, this leads to the 
standard eigenvalue problem Ay = Ay. But our matrix M will be tridiagonal, because 
neighboring hat functions overlap. It is exactly this situation that brings in the general- 
ized eigenvalue problem. The minimum value A, will be the smallest eigenvalue of 
Ay = AMy. That A, will be close to (and above) 2”. The eigenvector y will give the 
approximation U = y,V, +---+ynV, to the eigenfunction. 

As in the static problem. The method can be summarized in three steps: (1) choose 
the V;, (2) compute A and M, and (3) solve Ay = AMy. I don’t know why that costs a 
billion dollars. 


Rayleigh quotient R(v) 


Problem Set 6.5 


1. Use three hat functions, with h = }, to solve —u" = 2 with u(0) = u(1) =0. Verify 
that the approximation U matches u = x — x? at the nodes. 


2. Solve —u” = x with u(0) = u(1) = 0. Then solve approximately with two hat func- 
tions and h = }. Where is the largest error? 


3. Suppose —u” = 2, with the boundary condition u(1) = 0 changed to u/(1) = 0. This 
“natural” condition on uv’ need not be imposed on the trial functions V. With h = ; 
there is an extra half-hat V3, which goes from 0 to 1 between x = 5 and x = 1. 
Compute A33 = /{ (V3)?dx and f; = [2V3dx. Solve Ay = f for the finite element 


solution yj Vj + y2V2 + y3V3. 


4. Solve —u” =2 with a single hat function, but place its node at x = i instead of x = 5 


(Sketch this function V;.) With boundary conditions u(0) = u(1) = 0, compare the 


finite element approximation with the true u = x — x’. 


5. Galerkin’s method starts with the differential equation (say —u’ = f(x)) instead of 
the energy P. The trial solution is still uw = yj Vj + y2V2+---+ynVp, and the y’s are 
chosen to make the difference between —u” and f orthogonal to every V;: 


Galerkin / (—yi Vj! —yoV5' — +++ —yaVVidx = / f (x) Vj(x)dx. 
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integrate the left side by parts to reach Ay = f, proving that Galerkin gives the same 
A and f as Rayleigh-Ritz for symmetric problems. 


. A basic identity for quadratics shows y = A~!b as minimizing: 


1 1 _ = l 7 
Py a wy —y'b= 5 —A~'b)'A(y—A7!b) — xhA pb. 
The minimum over a subspace of trial functions is at the y nearest to A~'b. (That 
makes the first term on the right as small as possible; it is the key to convergence of 
U tou.) If A=/ and b = (1,0,0), which multiple of V = (1, 1,1) gives the smallest 
value of P(y) = 5y'y—y1? 


. Fora single hat function V(x) centered at x = 5, compute A = [(V’)?dx and M = 
{ Vdx. In the 1 by 1 eigenvalue problem, is A = A/M larger or smaller than the true 
eigenvalue A = 27? 


. For the hat functions V; and V> centered at x = h = 4 and x = 2h= , compute the 2 


by 2 mass matrix Mj; = { VjVjdx, and solve the eigenvalue problem Ax = AMx. 
1 


9. What is the mass matrix Mj; = { ViVjdx for n hat functions with h = ——? 


n+1° 


Computations with Matrices 


7.1. Introduction 


One aim of this book is to explain the useful parts of matrix theory. In comparison 
with older texts in abstract linear algebra, the underlying theory has not been radically 
changed. One of the best things about the subject is that the theory is really essential 
for the applications. What is different is the change in emphasis which comes with a 
new point of view. Elimination becomes more than just a way to find a basis for the 
row space, and the Gram-Schmidt process is not just a proof that every subspace has an 
orthonormal basis. Instead, we really need these algorithms. And we need a convenient 
description, A = LU or A = QR, of what they do. 

This chapter will take a few more steps in the same direction. I suppose these steps are 
governed by computational necessity, rather than by elegance, and I don’t know whether 
to apologize for that; it makes them sound very superficial, and that is wrong. They deal 
with the oldest and most fundamental problems of the subject, Ax = b and Ax = Ax, but 
they are continually changing and improving. In numerical analysis there is a survival 
of the fittest, and we want to describe some ideas that have survived so far. They fall 
into three groups: 


1. Techniques for Solving Ax = b. — Elimination is a perfect algorithm, except 
when the particular problem has special properties—as almost every problem has. Sec- 
tion 7.4 will concentrate on the property of sparseness, when most of the entries in A 
are zero. We develop iterative rather than direct methods for solving Ax = b. An iter- 
ative method is “self-correcting,” and never reaches the exact answer. The object is to 
get close more quickly than elimination. In some problems, that can be done; in many 
others, elimination is safer and faster if it takes advantage of the zeros. The competition 
is far from over, and we will identify the spectral radius that controls the speed of con- 
vergence tox = A~!b. 


2. Techniques for Solving Ax = Ax. The eigenvalue problem is one of the out- 
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standing successes of numerical analysis. It is clearly defined, its importance is obvious, 
but until recently no one knew how to solve it. Dozens of algorithms have been sug- 
gested, and everything depends on the size and the properties of A (and on the number 
of eigenvalues that are wanted). You can ask LAPACK for an eigenvalue subroutine, 
without knowing its contents, but it is better to know. We have chosen two or three ideas 
that have superseded almost all of their predecessors: the QR algorithm, the family of 
“power methods,’ and the preprocessing of a symmetric matrix to make it tridiagonal. 

The first two methods are iterative, and the last is direct. It does its job in a finite 
number of steps, but it does not end up with the eigenvalues themselves. This produces 
a much simpler matrix to use in the iterative steps. 


3. The Condition Number of a Matrix. Section 7.2 attempts to measure the 
“sensitivity” of a problem: If A and b are slightly changed, how great is the effect on 
x = A7~'b? Before starting on that question, we need a way to measure A and the change 
AA. The length of a vector is already defined, and now we need the norm of a matrix. 
Then the condition number, and the sensitivity of A will follow from multiplying the 
norms of A and A~!. The matrices in this chapter are square. 


7.2 Matrix Norm and Condition Number 


An error and a blunder are very different things. An error is a small mistake, probably 
unavoidable even by a perfect mathematician or a perfect computer. A blunder is much 
more serious, and larger by at least an order of magnitude. When the computer rounds 
oft a number after 16 bits, that is an error, But when a problem is so excruciatingly 
sensitive that this roundoff error completely changes the solution, then almost certainly 
someone has committed a blunder. Our goal in this section is to analyze the effect of 
errors, so that blunders can be avoided. 
We are actually continuing a discussion that began in Chapter 1 with 


ies 1 1 and B= 0.0001 1 
1 1.0001 1 1 


We claimed that B is well-conditioned, and not particularly sensitive to roundoff—except 
that if Gaussian elimination is applied in a stupid way, the matrix becomes completely 
vulnerable. It is a blunder to accept .0001 as the first pivot, and we must insist on a larger 
and safer choice by exchanging the rows of B. When “partial pivoting” is built into the 
elimination algorithm, the computer automatically looks for the largest pivot. Then the 
natural resistance to roundoff error is no longer compromised. 

How do we measure this natural resistance, and decide whether a matrix is well- 
conditioned or ill-conditioned? If there is a small change in D or in A, how large a 
change does that produce in the solution x? 
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We begin with a change in the right-hand side, from b to b+ 6b. This error might 
come from experimental data or from roundoff. We may suppose that db is small, but 
its direction is outside our control. The solution is changed from x to x+ 6x: 


Error equation  A(x+6x)=b+6b, so, by subtraction A(dx)=6b. (1) 


An error 5b leads to 5x = A~! 5b. There will be a large change in the solution x when 
A~! is large—A is nearly singular. The change in x is especially large when 5b points in 
the direction that is amplified most by A~!. 

Suppose A is symmetric and its eigenvalues are positive: 0 < Ay <---<A,. Any 


vector 6b is a combination of the corresponding unit eigenvectors x1,...,X,. The worst 
error 6x, coming from A~|, is in the direction of the first eigenvector x: 
Ob 
Worst error If 6b=ex,, then dx= 7" (2) 
I 


The error ||Sb|| is amplified by 1/41, which is the largest eigenvalue of A~'. This 
amplification is greatest when A, is near zero, and A is nearly singular. 

Measuring sensitivity entirely by A; has a serious drawback. Suppose we multiply all 
the entries of A by 1000; then A, will be multiplied by 1000 and the matrix will look 
much less singular. This offends our sense of fair play; such a simple rescaling cannot 
make an ill-conditioned matrix well. It is true that 6x will be 1000 times smaller, but so 
will the solution x = A~'b. The relative error ||6x||/||x|| will be the same. Dividing by 
||x|| normalizes the problem against a trivial change of scale. At the same time there is 
a normalization for 6b; our problem is to compare the relative change ||db||/||b|| with 
the relative error ||6x|| /||x||. 

The worst case is when ||6x|| is large—with 6b in the direction of the eigenvector 
x;—and when ||x|| is small. The true solution x should be as small as possible compared 
to the true b. This means that the original problem Ax = b should be at the other extreme, 
in the direction of the last eigenvector x,: if b = xy, then x =A~'b = b/Ay. 

It is this combination, b = x, and 6b = €x,, that makes the relative error as large as 
possible. These are the extreme cases in the following inequalities: 


7A Fora positive definite matrix, the solution x = A~'b and the error 6x = 
A~'65b always satisfy 


a 


|x| >4—- and psx) < HOE ang WORL — Anmax (15 


Amin I|x|] ~ Amin [JBI] 
The ratio c = Amax/Amin is the condition number of a positive definite matrix 
A. 


(3) 


Example 1. The eigenvalues of A are approximately A; = 10~4/2 and Az =2: 


1 1 
A= has condition number about c—4- 10%. 
1 1.0001 
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We must expect a violent change in the solution from ordinary changes in the data. 
Chapter 1 compared the equations Ax = b and Ax’ = b’: 


u + pS s-2 u + v=2 
u + 1.000lv = 2 u + 1.000lv = 2.0001. 


The right-hand sides are changed only by ||6b|| = .0001 = 10~*. At the same time, the 
solution goes from u = 2, v=0 tou =v= 1. This is a relative error of 

Sx _ (L,I _ v2 9. 1g lSell 

Ix (2,0) 2” l]>ll 
Without having made any special choice of the perturbation, there was a relatively large 
change in the solution. Our x and 6b make 45° angles with the worst cases, which 
accounts for the missing 2 between 2- 10* and the extreme possibility c = 4-104. 


which equals 


If A =/ or even if A =//10, its condition number is c = Amax/Amin = 1. By compari- 
son, the determinant is a terrible measure of ill-conditioning. It depends not only on the 
scaling but also on the order n; if A = 1/10, then the determinant of A is 10~”. In fact, 
this “nearly singular” matrix is as well-conditioned as possible. 


Example 2. The n by n finite difference matrix A has Amax © 4 and Amin & 17 /n?: 


—l 
=k 2 


The condition number is approximately c(A) = zn’, and this time the dependence on the 


order n is genuine. The better we approximate —u” = f, by increasing the number of 
unknowns, the harder it is to compute the approximation. At a certain crossover point, 
an increase in n will actually produce a poorer answer. 

Fortunately for the engineer, this crossover occurs where the accuracy is already 
pretty good. Working in single precision, a typical computer might make roundoff errors 
of order 10~?. With n = 100 unknowns and c = 5000, the error is amplified at most to 
be of order 10~°—which is still more accurate than any ordinary measurements. But 
there will be trouble with 10,000 unknowns, or with a 1, —4, 6, —4, 1 approximation to 


d*u/dx* = f(x)—for which the condition number grows as n*.! 


Unsymmetric Matrices 


Our analysis so far has applied to symmetric matrices with positive eigenvalues. We 
could easily drop the positivity assumption, and use absolute values |A|. But to go 


'The usual rule of thumb, experimentally verified, is that the computer can lose logc decimal places to the 
roundoff errors in Gaussian elimination. 
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beyond symmetry, as we certainly want to do, there will have to be a major change. This 
is easy to see for the very unsymmetric matrices 


[1 
pe and A l= ail (4) 
0 1 0 1 


Ps 


The eigenvalues all equal one, but the proper condition number is not Amax/Amin = |. 
The relative change in x is not bounded by the relative change in b. Compare 


x= , when b= n00 : x= re when JD’ = me 
1 1 0 0 


A 1% change in b has produced a hundredfold change in x; the amplification factor is 
100°. Since c represents an upper bound, the condition number must be at least 10,000. 
The difficulty here is that a large off-diagonal entry in A means an equally large entry in 
A~—!. Expecting A~! to get smaller as A gets bigger is often wrong. 

For a proper definition of the condition number, we look back at equation (3). We 
were trying to make x small and b = Ax large. When A is not symmetric, the maximum of 
||Ax||/||x|| may be found at a vector x that is not one of the eigenvectors. This maximum 
is an excellent measure of the size of A. It is the norm of A. 


7B The norm of A is the number ||A|| defined by 


Ax 
[Al] = max aT 5) 
x40 ||| 
In other words, ||A|| bounds the “amplifying power” of the matrix: 
|Ax|| < ||A|] |||] for all vectors x. (6) 


The matrices A and A~! in equation (4) have norms somewhere between 100 and 101. 
They can be calculated exactly, but first we want to complete the connection between 
norms and condition numbers. Because b = Ax and 6x = A~! 6b, equation (6) gives 


>] <Alllxl] = and |] 5x|| < Am" [80]. (7) 


This is the replacement for equation (3), when A is not symmetric. In the symmetric case, 
||A|| is the same as Amax, and ||A~'|| is the same as 1/Amin. The correct replacement for 
Amax/ Amin is the product ||A\|||A~'||—-which is the condition number. 


7C The condition number of A is c = ||A\|||A~'||. The relative error satisfies 


) ob 
6x from 6b ; : es ; 7 directly from equation (7). (8) 
x 
If we perturb the matrix A instead of the right-hand side b, then 
) oA 
6x from dA alhOAN < Zll@All from equation (10) below. (9) 


l|x+ Ox|| ~ All| 
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What is remarkable is that the same condition number appears in equation (9), when the 
matrix itself is perturbed: If Ax = b and (A + 6A)(x+ 6x) = b, then by subtraction 


Aédx+ dA(x+ 6x) =0, or dx = —A~!(6A)(x+ 5x). 


Multiplying by 6A amplifies a vector by no more than ||6A||, and multiplying by A7! 


amplifies by no more than ||A~!||. Then ||6x|| < ||A~!]|||6A]|||x-+ 6x||, which is 
|| ox = || 5Al| 
eel < Aq I18Al] = ete. (10) 
Ix + d2| |All 


These inequalities mean that roundoff error comes from two sources. One is the 
natural sensitivity of the problem, measured by c. The other is the actual error 6b 
or 6A. This was the basis of Wilkinson’s error analysis. Since elimination actually 
produces approximate factors L’ and U’, it solves the equation with the wrong matrix 
A+ 6A = L'U’ instead of the right matrix A = LU. He proved that partial pivoting 
controls 6A—so the burden of the roundoff error is carried by the condition number c. 


A Formula for the Norm 


The norm of A measures the largest amount by which any vector (eigenvector or not) 
is amplified by matrix multiplication: ||A|| = max(||Ax|| /||x||). The norm of the identity 
matrix is 1. To compute the norm, square both sides to reach the symmetric A'A: 
Ax||* TATA 
(Almas! I = max. (11) 
[| xx 
7D ||A|j is the square root of the largest eigenvalue of ATA: ||A||? = Amax(A‘A). 
The vector that A amplifies the most is the corresponding eigenvector of ATA: 
x! ATAx = a (Aaa) 
T 


= Amax(A™A) = ||A|l?. (12) 


ne 
Figure 7.1 shows an unsymmetric matrix with eigenvalues A; = Az = 1 and norm 
||A|| = 1.618. In this case A~! has the same norm. The farthest and closest points Ax on 


the ellipse come from eigenvectors x of A'A, not of A. 


Note 1. The norm and condition number are not actually computed in practice, only 
estimated, There is not time to solve an eigenvalue problem for Amax(A'A). 


Note 2. In the least-squares equation A'Ax = A‘), the condition number c(A‘A) is the 
square of c(A). Forming A'A can turn a healthy problem into a sick one. It may be 
necessary to orthogonalize A by Gram-Schmidt, instead of computing with A'A. 


Note 3. The singular values of A in the SVD are the square roots of the eigenvalues of 
A'A. By equation (12), another formula for the norm is ||A|] = Omax. The orthogonal 
U and V leave lengths unchanged in ||Ax|| = ||UZV'x||. So the largest ||Ax|| /||x|| comes 
from the largest o in the diagonal matrix ©. 
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sae 1+ V5 
- = = 
0 : |All ; 
me : 
gn dk £2 I|All = Xmax(A A) & 2.618 
1 
Tal Amin ATA ~ 0.382 
|4—"]/2 ( ) 
c(A) = |[Al|||A7*]] © (1.618)? 
ellipse of all Ax 
circle ||x|| = 1 


Figure 7.1: The norms of A and A~! come from the longest and shortest Ax. 


Note 4, Roundoff error also enters Ax = Ax. What is the condition number of the eigen- 
value problem? The condition number of the diagonalizing S measures the sensitivity 
of the eigenvalues. If wu is an eigenvalue of A+ E, then its distance from one of the 
eigenvalues of A is 


JHA] < |ISII|S~ "El = e(S) |Z. (13) 


With orthonormal eigenvectors and S$ = Q, the eigenvalue problem is perfectly condi- 
tioned: c(Q) = 1. The change 6A in the eigenvalues is no greater than the change 6A. 
Therefore the best case is when A is symmetric, or more generally when AAT = ATA. 
Then A is a normal matrix; its diagonalizing S is an orthogonal Q (Section 5.6). 

If x; is the kth column of § and y, is the kth row of S~!, then A, changes by 


OA, = y, Ex, + terms of order |E ||. (14) 


In practice, y,E x, is a realistic estimate of 6A. The idea in every good algorithm is to 
keep the error matrix E as small as possible—usually by insisting, as in the next section, 
on orthogonal matrices at every step of the computation of /. 


Problem Set 7.2 


1. For an orthogonal matrix Q, show that ||Q|| = 1 and also c(Q) = 1. Orthogonal 
matrices (and their multiples @Q) are the only perfectly conditioned matrices. 


2. Which “famous” inequality gives ||(A+B)x|| < ||Ax|| +||/Bx 
from equation (5) that ||A + B|| < ||A]| + ||B]]? 


, and why does it follow 


3. Explain why ||ABx|| < ||A||||B]| |||, and deduce from equation (5) that ||AB|| < ||A||||B|]. 
Show that this also implies c(AB) < c(A)c(B). 


10. 


11. 


12. 


13. 


14. 
15. 


16. 
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For the positive definite A = | *, 5'], compute ||A~"|| = 1/4), ||A|| = As, and c(A) = 
Az/A,. Find a right-hand side b and a perturbation 6b so that the error is the worst 
possible, ||6.x||/||x|| = ¢|[65]|/||5]]. 


Show that if A is any eigenvalue of A, Ax = Ax, then |A| < ||Al|. 
The matrices in equation (4) have norms between 100 and 101. Why? 
Comparing the eigenvalues of ATA and AA!, prove that ||A|| = ||A‘]. 


For a positive definite A, the Cholesky decomposition is A = LDL' = R'™R, where 
R= DL". Show directly from equation (12) that the condition number of c(R) is 
the square root of c(A). Elimination without row exchanges cannot hurt a positive 
definite matrix, since c(A) = c(R")c(R). 


Show that max |A| is not a true norm, by finding 2 by 2 counterexamples to Amax(A + 
B) < Amax(A) + Amax(B) and Amax(AB) < Amax(A)Amax (B). 


Show that the eigenvalues of B = [ (i 
B. 


‘ are +0j, the singular values of A. Hint: Try 
(a) Do A and A~! have the same condition number c? 
(b) In parallel with the upper bound (8) on the error, prove a lower bound: 


]Oxl] . 1 1]6dI| 


= : (Consider A~'b =x instead of Ax = b.) 
Ilxl| ~ © [JBI] 


Find the norms Amax and condition numbers Amax/Amin Of these positive definite 


matrices: 
100 O 2 It 2 a | 
0 2 b) 2 jes 


Find the norms and condition numbers from the square roots of Amax(ATA) and 


Amin(ATA): 
=2°0 1 1 1 1 
0 2 00 ==) en 


Prove that the condition number ||A||||A~'|j is at least 1. 


Why is J the only symmetric positive definite matrix that has Amax = Amin = 1? Then 
the only matrices with ||A|| = 1 and ||A~'|| = 1 must have ATA = J. They are 
matrices. 


Orthogonal matrices have norm ||Q|| = 1. If A = QR, show that ||A|| < ||R|| and also 
||R|| < ||A|]. Then ||A]| = ||Q]|||R]]. Find an example of A = LU with ||A|| < ||Z]|||U]]. 
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17. (Suggested by Moler and Van Loan) Compute b — Ay and b— Az when 
AIS ol 
ies i 80.563 
254 913.659 
Is y closer than z to solving Ax = b? Answer in two ways: Compare the residual 


b — Ay to b— Az. Then compare y and z to the true x = (1,—1), Sometimes we want 
a small residual, sometimes a small 6x. 


Problems 18-20 are about vector norms other than the usual ||x|| = \/x-x. 


18. The “¢! norm” is |x|]; = |x|] -+---+|x|n. The “2” norm” is ||x||.. = max |2x;|. Compute 
\|x||, ||x|]1 and ||x||.. for the vectors 


> 


x=(1,1,1,1,1) and = x=(.1,.7,.3,.4,.5). 


19. Prove that ||x||.. < ||x|| < ||x||1. Show from the Schwarz inequality that the ratios 
\|x]| / |||]. and ||x||1/|lx|| are never larger than \/n. Which vector (x1,...,%,) gives 
ratios equal to \/n? 


20. All vector norms must satisfy the triangle inequality. Prove that 


IX Ylloo S [lI oot [lylleo and |x +ylfa S [lll + [ly lh. 


21. Compute the exact inverse of the Hilbert matrix A by elimination. Then compute 
A~! again by rounding all numbers to three figures: 


In MATLAB: A =hilb(3) = 


WIRNIR — 
BIR WIR NIE 
le BI] Re Go| 


22. For the same A, compute b = Ax for x = (1, 1,1) and x = (0,6, —3.6). A small change 
Ab produces a large change Ax. 


23. Compute Amax and Amin for the 8 by 8 Hilbert matrix aj; = 1/(i+ j—1). If Ax =b 
with ||b|| = 1, how large can ||x|| be? If b has roundoff error less than 10~!°, how 
large an error can this cause in x? 


24. If you know L, U, Q, and R, is it faster to solve LUx = b or QRx = b? 


25. Choosing the largest available pivot in each column (partial pivoting), factor each A 
into PA = LU: 


1 0 
pao. 


101 
A= | and A=12 2 0 
02 0 
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26. Find the LU factorization of A = bi re On your computer, solve by elimination 
when € = 1077, 10-°, 10-?, 10-!7, 10- >: 


ce 


The true x is (1,1). Make a table to show the error for each €. Exchange the two 
equations and solve again—the errors should almost disappear. 


Le 
2 


7.3. Computation of Eigenvalues 


There is no one best way to find the eigenvalues of a matrix. But there are certainly 
some terrible ways which should never be tried, and also some ideas that do deserve a 
permanent place. We begin by describing one very rough and ready approach, the power 
method, whose convergence properties are easy to understand. We added a graphic 
animation (with sound) to the course page web. mit.edu/18.06, to show the power method 
in action. 

We move steadily toward a more sophisticated algorithm, which starts by making a 
symmetric matrix tridiagonal and ends by making it virtually diagonal. That second step 
is done by repeating Gram-Schmidt, so it is known as the QR method. 

The ordinary power method operates on the principle of a difference equation. It 
starts with an initial guess ug and then successively forms uj = Aug, uz = Auy, and 
in general uz,; = Aug. Each step is a matrix-vector multiplication. After k steps it 
produces uj = A*ug, although the matrix A‘ will never appear. The essential thing is 
that multiplication by A should be easy—if the matrix is large, it had better be sparse— 
because convergence to the eigenvector is often very slow. Assuming A has a full set of 
eigenvectors x1,...,Xn, the vector uz will be given by the usual formula: 


Eigenvectors weighted by A‘ a cyAkxy fees CnA ken. 


Suppose the largest eigenvalue A, is all by itself; there is no other eigenvalue of the same 
magnitude, and |A;| < --- < |A,_1| < |A,|. Then as long as the initial guess ug contained 
some component of the eigenvector x,, so that c, 4 0, this component will gradually 
dominate in ux: 


Ai \* RAN 
qe =e (3) MG (=) Xn—-1 + CnXn- (1) 


The vectors uz point more and more accurately toward the direction of X,. Their conver- 
gence factor is the ratio r = |Ay—,|/|An|. It is just like convergence to a steady state, for 
a Markov matrix, except now A, may not equal 1. The scaling factor A* in equation (1) 
prevents u; from growing very large or very small, in case |A,,| > 1 or |Ay| < 1. 
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Often we can just divide each u, by its first component O; before taking the next step. 
With this simple scaling, the power method uz,; = Au, /O,% converges to a multiple of 
Xn. The scaling factors oj, will approach Ay. 


Example 1. The uw, approach the eigenvector fal = [$87] when A = |-? 2] is the 


matrix of population shifts in Section 1.3: 


1 9 83 781 | 747 
— uy = Ww = le = . 
ea he el Bek | ia)? AT oil 2 4= | 253 


If r = |An_1|/|An| is close to 1, then convergence is very slow. In many applications 
r > .9, which means that more than 20 iterations are needed to achieve one more digit. 
(The example had r = .7, and it was still slow.) If r= 1, which means |A,_| = |A,|, then 
convergence will probably not occur at all. That happens (in the applet with sound) for a 
complex conjugate pair A,_; =A,. There are several ways to get around this limitation, 
and we shall describe three of them: 


1. The block power method works with several vectors at once, in place of uz. If we 
multiply p orthonormal vectors by A, and then apply Gram-Schmidt to orthogonal- 
ize them again—that is a single step of the method—the convergence ratio becomes 
1 = |An—p|/|An|. We will obtain approximations to p different eigenvalues and their 
elgenvectors. 


2. The inverse power method operates with A~! instead of A. A single step is v4.1 = 
A~!v,, which means that we solve the linear system Av;41 =v, (and save the factors 
L and U!). Now we converge to the smallest eigenvalue 4, and its eigenvector x1, 
provided |A,| < |A,|. Often it is A; that is wanted in the applications, and then 
inverse iteration is an automatic choice. 


3. The shifted inverse power method is best of all. Replace A by A — a/. Each eigen- 
value is shifted by @, and the convergence factor for the inverse method will change 
to r” = |A, — a|/|Azg — @|. If & is a good approximation to A, 7” will be very small 
and the convergence is enormously accelerated. Each step of the method solves 
(A = Ol)we+1 = Wk: 

C{X] C2X2 CnXn 


(—a)k (a-ak Ta) 


Wy = 


When @ is close to A;, the first term dominates after only one or two steps. If 
A, has already been computed by another algorithm (such as QR), then @ is this 
computed value. One standard procedure is to factor A — a@/ into LU and to solve 
Ux, = (1,1,...,1) by back-substitution. 


If A; is not already approximated, the shifted inverse power method has to generate its 
own choice of a. We can vary & = O% at every step if we want to, so (A — O41 )wey1 = We. 
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When A is symmetric, a very accurate choice is the Rayleigh quotient: 


wi Aw, 


T . 
W,Wk 


shift by O% = R(wx) = 


This quotient R(x) has a minimum at the true eigenvector x,. Its graph is like the bottom 
of a parabola, so the error A; — Oy is roughly the square of the error in the eigenvector. 
The convergence factors |A; — o%|/|A2 — o%| are themselves converging to zero. Then 
these Rayleigh quotient shifts give cubic convergence of 0%, to Ay.” 


Tridiagonal and Hessenberg Forms 


The power method is reasonable only for a matrix that is large and sparse. When too 
many entries are nonzero, this method is a mistake. Therefore we ask whether there is 
any simple way fo create zeros. That is the goal of the following paragraphs. 

It should be said that after computing a similar matrix Q~'AQ with more zeros than 
A, we do not intend to go back to the power method. There are much more powerful 
variants, and the best of them seems to be the QR algorithm. (The shifted inverse power 
method has its place at the very end, in finding the eigenvector.) The first step is to pro- 
duce quickly as many zeros as possible, using an orthogonal matrix Q. If A is symmetric, 
then so is Q~'AQ. No entry can become dangerously large because Q preserves lengths. 

To go from A to Q~'AQ, there are two main possibilities: We can produce one zero at 
every step (as in elimination), or we can work with a whole column at once. For a single 
zero, it is easy to use a plane rotation as illustrated in equation (7), found near the end 
of this section, that has cos @ and sin @ in a 2 by 2 block. Then we could cycle through 
all the entries below the diagonal, choosing at each step a rotation @ that will produce a 
zero; this is Jacobi’s method. It fails to diagonalize A after a finite number of rotations, 
since the zeros from early steps will be destroyed when later zeros are created. 

To preserve the zeros and stop, we have to settle for less than a triangular form. 
The Hessenberg form accepts one nonzero diagonal below the main diagonal. If a 
Hessenberg matrix is symmetric, it only has three nonzero diagonals. 

A series of rotations in the right planes will produce the required zeros. Householder 
found a new way to accomplish exactly the same thing. A Householder transformation 
is a reflection matrix determined by one vector v: 


T 


Householder matrix H=I- aes 
v 


Often v is normalized to become a unit vector u = v/||v||, and then H becomes J — 2uu'. 


In either case H is both symmetric and orthogonal: 


H'H= (I- 2uu') (I — 2uu') = 7 — Ayu! +4uu' uu! = 1. 


?Linear convergence means that every step multiplies the error by a fixed factor r < 1. Quadratic convergence 
means that the error is squared at every step, as in Newton’s method xy) — x, = —f (xx) /f' (xg) for solving f(x) = 
0. Cubic convergence takes 10! to 1077 to 107°. 
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Thus H = H' = H~!. Householder’s plan was to produce zeros with these matrices, and 
its success depends on the following identity Hx = —0z: 


7E Suppose z is the column vector (1,0,...,0), o = ||x||, and v = x+ oz. 
Then Hx = —oz = (—o,0,...,0). The vector Hx ends in zeros as desired. 


The proof is to compute Hx and reach — oz: 


2vvT 2(x+0z)' 
Hx=x— w= x—(x+02) oe Je 
I|v| (x+ 0z)!(x+ 02) (2) 
=x—(x+0z) (because x! x = 07) 
= —Oz. 


This identity can be used right away, on the first column of A. The final Q~!AQ is 
allowed one nonzero diagonal below the main diagonal (Hessenberg form). Therefore 
only the entries strictly below the diagonal will be involved: 


ar 1 —O 
a3 0 0 

BS Ve Wg BS hols Ax=| . |. (3) 
an\ 0 0 


At this point Householder’s matrix H is only of order n — 1, so it is embedded into the 
lower right-hand corner of a full-size matrix U}: 


10 0 0 0 aij * * * x 
0 —-O * * * x 
Uo. OF =U;', and U,'AU=| 0 * * * * 
0 Qi see ok” 
0 O « * x x 


The first stage is complete, and U, ‘AU, has the required first column. At the second 
stage, x consists of the last n — 2 entries in the second column (three bold stars). Then 
A is of order n— 2. When it is embedded in Up, it produces 


100 0 0 cea oR oe 
010 0 0 * * * Ok Xx 
Ur=|0 0 =U;!,  Uy'(Up!AU)U,= |0 * * * * 
0 0 eb) 00x x* x 
0 0 00 «x * x 


U3 will take care of the third column. For a 5 by 5 matrix, the Hessenberg form is 
achieved (it has six zeros). In general Q is the product of all the matrices U;U2---Uy_2, 
and the number of operations required to compute it is of order n°. 
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Example 2. (to change a13 = a3 to zero) 


10 1 
A= ]|0 1 1], x= 
1 1 0 


Embedding H into Q, the result Q~!AQ is tridiagonal: 


L3G." 6 1 -1 0 
Q=|0 0 -i|, Q'AQ=|-1 0 1 
0: 2h “9 0 141 


Q-'AQ is a matrix that is ready to reveal its eigenvalues—the OR algorithm is ready to 
begin—but we digress for a moment to mention two other applications of these same 
Householder matrices H. 


1. The Gram-Schmidt factorization A = QR. Remember that R is to be upper trian- 
gular. We no longer have to accept an extra nonzero diagonal below the main one, 
since no matrices are multiplying on the right to spoil the zeros. The first step in 
constructing Q is to work with the whole first column of A: 


a\1 1 

a2} 0 vl 
x=]. |, oe Be v=x+t ||x|[z, te 

An 0 


The first column of H)A equals —||x||z. It is zero below the main diagonal, and it 
is the first column of R. The second step works with the second column of H,A, 
from the pivot on down, and produces an H2HjA which is zero below that pivot. 
(The whole algorithm is like elimination, but slightly slower.) The result of n — 1 
steps is an upper triangular R, but the matrix that records the steps is not a lower 
triangular L. Instead it is the product Q = H,H2---H,—1, which can be stored in this 
factored form (keep only the v’s) and never computed explicitly. That completes 
Gram-Schmidt. 


2. The singular value decomposition U'AV =X. The diagonal matrix © has the same 
shape as A, and its entries (the singular values) are the square roots of the eigenval- 
ues of AA. Since Householder transformations can only prepare for the eigenvalue 
problem, we cannot expect them to produce ©. Instead, they stably produce a bidi- 
agonal matrix, with zeros everywhere except along the main diagonal and the one 
above. 


The first step toward the SVD is exactly as in QR above: x is the first column of A, 
and Hx is zero below the pivot. The next step is to multiply on the right by an H) 
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which will produce zeros as indicated along the first row: 


* * * x * *« 0 0 
ARH A= 0. &. 3-4) IGA = 10. ee] (4) 
O * * x O «x «x x 


Then two final Householder transformations quickly achieve the bidiagonal form: 


* * 0 0 * * 0 0 
HoH|AH = |0 « * and = ApH AHH?) = 10 « « 0 
0 O x« 00x 


The QR Algorithm for Computing Eigenvalues 


The algorithm is almost magically simple. It starts with Ao, factors it by Gram-Schmidt 
into QoRo, and then reverses the factors: Aj = RoQo. This new matrix A is similar to 
the original one because Q5 "AQo =Q) ! (QoRo)Qo = Ai. So the process continues with 
no change in the eigenvalues: 


All A; are similar A, = Q,R, and then Agit = RQ. (5) 


This equation describes the unshifted QR algorithm, and almost always Ax; approaches a 
triangular form, Its diagonal entries approach its eigenvalues, which are also the eigen- 
values of Ao. If there was already some processing to obtain a tridiagonal form, then Ag 
is connected to the absolutely original A by Q~'AQ = Apo. 

As it stands, the QR algorithm is good but not very good. To make it special, it needs 
two refinements: We must allow shifts to A; — a,/, and we must ensure that the OR 
factorization at each step is very quick. 


1. The Shifted Algorithm. If the number a, is close to an eigenvalue, the step in 
equation (5) should be shifted immediately by a; (which changes Q; and R;,): 


Ax = Ol = OR; and then Agyt = ReQgE + Ol. (6) 
This matrix A;,, is similar to A; (always the same eigenvalues): 
Oy 'AcQn = Oy" (QR + OL) On = Acs. 


What happens in practice is that the (n,n) entry of A,—the one in the lower right-hand 
corner—is the first to approach an eigenvalue. That entry is the simplest and most pop- 
ular choice for the shift o;,. Normally this produces quadratic convergence, and in the 
symmetric case even cubic convergence, to the smallest eigenvalue. After three or four 
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steps of the shifted algorithm, the matrix A; looks like this: 


Ay = with ¢€ <1. 


* Ox 
* Ox 
QO x 
0 0 


mM) x* x* * 
* 


We accept the computed A; as a very close approximation to the true A;. To find the 
next eigenvalue, the QR algorithm continues with the smaller matrix (3 by 3, in the 
illustration) in the upper left-hand corner. Its subdiagonal elements will be somewhat 
reduced by the first QR steps, and another two steps are sufficient to find A2. This gives 
a systematic procedure for finding all the eigenvalues. In fact, the QR method is now 
completely described. It only remains to catch up on the eigenvectors—that is a single 
inverse power step—and to use the zeros that Householder created. 


2. When Ag is tridiagonal or Hessenberg, each QR step is very fast. The Gram-Schmidt 
process (factoring into OR) takes O(n*) operations for a full matrix A. For a Hessenberg 
matrix this becomes O(n”), and for a tridiagonal matrix it is O(n). Fortunately, each new 
A; is again in Hessenberg or tridiagonal form: 


* 


oo x 


x* %X* * 


Qo isHessenberg Qo =AoRp' = 


Co * 
x¥ * %¥ * 
*¥ * %¥ * 
oo * * 
*¥ * %X * 


oS &* «+ + 


0 0 0 


You can easily check that this multiplication leaves Qo with the same three zeros as Ao. 
Hessenberg times triangular is Hessenberg. So is triangular times Hessenberg: 


* * *& *| [fk ok Ok Ox 
; OF) See ko ea GeO ae a ae 

A, is Hessenberg A; = RoQ0 = 
0 0 «x x!| 10 «x x x 
000 «| |0 0 x x 


The symmetric case is even better, since A} = Qo "A0Q0 = QOtA0Qo stays symmetric. By 
the reasoning just completed, A; is also Hessenberg. So A; must be tridiagonal. The 
same applies to A2,A3,..., and every QR step begins with a tridiagonal matrix. 

The last point is the factorization itself, producing the Q; and R; from each A, (or 
really from A; — oJ). We may use Householder again, but it is simpler to annihilate 
each subdiagonal element in turn by a “plane rotation” P;;. The first is P21: 


cos@ —sin@ ayy * * x 
in @ ) 
Rotation to kill a2, PA, = Se eo i i 7) 
1 Oo oe ee & 
1 0 O x x 
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The (2,1) entry in this product is aj; sin @ + a2; cos @, and we choose the angle @ that 
makes this combination zero. The next rotation P32 is chosen in a similar way, to remove 
the (3,2) entry of P32P),A,. After n — 1 rotations, we have Ro: 


Triangular factor Ry = Pan—1-++: P32 Po Ag. (8) 


Books on numerical linear algebra give more information about this remarkable algo- 
rithm in scientific computing. We mention one more method—Arnoldi in ARPACK— 
for large sparse matrices. It orthogonalizes the Krylov sequence x,Ax,A7x,... by Gram- 
Schmidt. If you need the eigenvalues of a large matrix, don’t use det(A — AJ)! 


Problem Set 7.3 


1. For the matrix A = is 5 | with eigenvalues A; = 1 and Az = 3, apply the power 


method uz; = Au, three times to the initial guess up = Ck What is the limiting 
vectOFr Uso? 


2. For the same A and the initial guess ug = [3] , compare three inverse power steps to 
one shifted step with @ = uj Aug / uUduo: 


1 


2 1 
Wii =A ‘uy = | 


l 2 


3 | Uk or u=(A—al)~luo. 


The limiting vector vu. is now a multiple of the other eigenvector (1,1). 


3. Explain why |A,/A,—1| controls the convergence of the usual power method. Con- 
struct a matrix A for which this method does not converge. 


4. The Markov matrix A = |‘? 3] has A = 1 and .6, and the power method uz = A*ug 
converges to els Find the eigenvectors of A~'. What does the inverse power 


method u_; = A~*uo converge to (after you multiply by .6*)? 


5. Show that for any two different vectors of the same length, ||x|| = ||y||, the House- 


holder transformation with v = x — y gives Hx = y and Hy = x. 


6. Compute o = ||x||, v=x+ oz, and H =I —2vv!/v"y, Verify Hx = —oz: 


10. 


11. 


12. 


13. 


14. 


15. 


16. 


7.4 
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Show that starting from Ap = fe | , the unshifted QR algorithm produces only the 


modest improvement A; = i eal 


Apply to the following matrix A a single QR step with the shift & = a22—which in 
this case means without shift, since a22 = 0. Show that the off-diagonal entries go 
from sin @ to —sin*@, which is cubic convergence. 


qe coe sin 0 
sin 0 0 


Check that the tridiagonal A = i 4 is left unchanged by the QR algorithm. It is one 
of the (rare) counterexamples to convergence (so we shift). 


Show by induction that, without shifts, (QoQ) ---Q,)(Rx---R1Ro) is exactly the QR 
factorization of Ay, ;. This identity connects QR to the power method and leads to 
an explanation of its convergence. If |A;| > |A2| >--- > |A,|, these eigenvalues will 
gradually appear on the main diagonal. 


Choose sin @ and cos @ in the rotation P to triangularize A, and find R: 
Bae ON8 —sin@| }1 —1 2 & 
sin@ cos@ |} |3 5 QO x 


Choose sin@ and cos @ to make Py AP3,' triangular (same A). What are the eigen- 
values? 


= Kh. 


When A is multiplied by P;; (plane rotation), which entries are changed? When P;;A 
is multiplied on the right by Bee which entries are changed now? 

How many multiplications and how many additions are used to compute PA? (A 
careful organization of all the rotations gives $n? multiplications and additions, the 
same as for QR by reflectors and twice as many as for LU.) 


(Turning a robot hand) A robot produces any 3 by 3 rotation A from plane rotations 
around the x, y, and z axes. If P32P3;P),;A = TJ, the three robot turns are in A = 
ja is a The three angles are Euler angles. Choose the first 0 so that 


cos@ —sin@ 0 Slt <2. -.22 
Px3A=1|sin@ cosd 0 5 2 -1 2 is zero in the (2,1) position. 
0 0 1 2 2 -1 


Iterative Methods for Ax = b 


In contrast to eigenvalues, for which there was no choice, we do not absolutely need 
an iterative method to solve Ax = b. Gaussian elimination will reach the solution x in 
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a finite number of steps (n°/3 for a full matrix, less than that for the large matrices we 
actually meet), Often that number is reasonable. When it is enormous, we may have to 
settle for an approximate x that can be obtained more quickly—and it is no use to go 
part way through elimination and then stop. 

Our goal is to describe methods that start from any initial guess xp, and produce an 
improved approximation x,+; from the previous x,;. We can stop when we want to. 

An iterative method is easy to invent, by splitting the matrix A. If A= S—T, then 
the equation Ax = b is the same as Sx = Tx +b. Therefore we can try 


Iteration from x; to x;,+1 Sxe41 = Tx, + bz (1) 


There is no guarantee that this method is any good. A successful splitting S — T satisfies 
two different requirements: 


1. The new vector x;,,, should be easy to compute. Therefore S should be a simple 
(and invertible!) matrix; it may be diagonal or triangular. 


2. The sequence x; should converge to the true solution x. If we subtract the iteration 
in equation (1) from the true equation Sx = Tx +b, the result is a formula involving 
only the errors e, = x — Xx: 


Error equation Sexi, = Tex. (2) 


This is just a difference equation. It starts with the initial error eo, and after k steps 
it produces the new error e, = (S~'T)‘ep. The question of convergence is exactly 
the same as the question of stability: x, — x exactly when e; — 0. 


7F The iterative method in equation (1) is convergent if and only if every 
eigenvalue of S~'T satisfies |A| < 1. Its rate of convergence depends on the 
maximum size of |A|: 


Spectral radius “rho” p(S-'T) = max|Aij. (3) 


Remember that a typical solution to e,4, = S~!Te, is a combination of eigenvectors: 
Error after k steps ep = CyAk xy ++ eg Aen. (4) 


The largest |A;| will eventually be dominant, so the spectral radius p = |Amax| will govern 
the rate at which e; converges to zero. We certainly need p < 1. 

Requirements | and 2 above are conflicting. We could achieve immediate conver- 
gence with S = A and T = 0; the first and only step of the iteration would be Ax; = b. In 
that case the error matrix S~!T is zero, its eigenvalues and spectral radius are zero, and 
the rate of convergence (usually defined as — log p) is infinite. But Ax; = b may be hard 
to solve; that was the reason for a splitting. A simple choice of S can often succeed, and 
we start with three possibilities: 
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1. S = diagonal part of A (Jacobi’s method). 
2. S = triangular pail of A (Gauss-Seidel method). 
3. S = combination of | and 2 (successive overrelaxation or SOR). 
S is also called a preconditioner, and its choice is crucial in numerical analysis. 


Example 1 (Jacobi). Here S is the diagonal part of A: 


2 -l 2 1 I 
S= T= » S'T= : 2]. 

—1 2 2 1 0 x 0 
If the components of x are v and w, the Jacobi step Sx,41 = Tx, +b is 

2VE41 = Wetdy v 0 5 v b,/2 

We = Ve tb cane Vereen | by /2| 

Wk+1 = Vk T 92, eid 5 Wa #2 

The decisive matrix S~'T has eigenvalues +3, which means that the error is cut in half 


(one more binary digit becomes correct) at every step. In this example, which is much 
too small to be typical, the convergence is fast. 


7 


For a larger matrix A, there is a very practical difficulty. The Jacobi iteration re- 
quires us to keep all components of x; until the calculation of x;,,; is complete. A 
much more natural idea, which requires only half as much storage, is to start using each 
component of the new x;,1 as soon as it is computed; x;,,, takes the place of x, a com- 
ponent at a time. Then x; can be destroyed as fast as x,1 is created, The first component 
remains as before: 


New x, = ay (x1)e41 = (—@12%2 — 413"3 — +++ — GinXn)e t+ D1. 
The next step operates immediately with this new value of x,, to find (x2),41: 
New x2 70(*2)e41 = —@ai(%1)e41 + (—€23%3 — +++ — AanXn)xt bo. 
And the last equation in the iteration step will use new values exclusively: 
New xX; = Ann(Xn)e+1 = (Gn X1 — An2X2 — +++ — Ann—1%n—1) 41 + On: 


This is called the Gauss-Seidel method, even though it was apparently unknown to 
Gauss and not recommended by Seidel. That is a surprising bit of history, because it is 
not a bad method. When the terms in x,,1 are moved to the left-hand side, S is seen as 
the lower triangular part of A. On the right-hand side, T is strictly upper triangular. 


Example 2 (Gauss-Seidel). Here S~'!T has smaller eigenvalues: 


ie eC pec, sete. (0 
“4D 00 0 


ae ee 


AIF NIE 
| | 
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A single Gauss-Seidel step takes the components v; and w, into 


2Ve41 =Wetdy ey 2 0O 
QW = Veto, 


0 1 
XG — 
k+1 0 0 


The eigenvalues of S~'T are r and 0. The error is divided by 4 every time, so a sin- 
gle Gauss-Seidel step is worth two Jacobi steps. Since both methods require the same 
number of operations—we just use the new value instead of the old, and actually save 
on storage—the Gauss-Seidel method is better. 

This rule holds in many applications, even though there are examples in which Jacobi 
converges and Gauss-Seidel fails (or conversely). The symmetric case is straightfor- 
ward: When all aj; > 0, Gauss-Seidel converges if and only if A is positive definite. 


It was discovered during the years of hand computation (probably by accident) that 
convergence is faster if we go beyond the Gauss-Seidel correction x44 — xz. Roughly 
speaking, those approximations stay on the same side of the solution x. An overrelax- 
ation factor ® moves us closer to the solution. With @ = 1, we recover Gauss-Seidel; 
with @ > 1, the method is known as successive overrelaxation (SOR). The optimal 
choice of @ never exceeds 2. It is often in the neighborhood of 1.9. 

To describe overrelaxation, let D, L, and U be the parts of A on, below, and above 
the diagonal, respectively. (This splitting has nothing to do with the A = LDU of elim- 
ination. In fact we now have A= L+D+U.) The Jacobi method has S = D on the 
left-hand side and T = —L—U on the right-hand side. Gauss-Seidel chose S = D+ L 
and T = —U. To accelerate the convergence, we move to 


Overrelaxation [D+ @L)xx41 = [1 — @)D— OU |x, + ob. (5) 


Regardless of @, the matrix on the left is lower triangular and the one on the right is 
upper triangular. Therefore x, can still replace x,, component by component, as soon 
as it is computed. A typical step is 


ii (Xi) k= Gi (xi), + @[(—ayx1 — +++ — G—19%;—-1) R41 + (— ix; — +++ — GinXn) ge + Di). 


If the old guess x, happened to coincide with the true solution x, then the new guess x; 
would stay the same, and the quantity in brackets would vanish. 


Example 3 (SOR). For the same A = ee ale each overrelaxation step is 


| 2 ; Pee @ 
Xk = 


ab. 
Ho 2 0 20-e)|** 


If we divide by @, these two matrices are the S and T in the splitting AS — 7; the iteration 
is back to Sxg4, = Txz +b. The crucial matrix L = S~'T is 


allies 0] 2(1—@) o | | 1-@ 50 
a er ae 0 2(1-@)| |4@(1-@) 1-@+1@? 
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The optimal @ makes the largest eigenvalue of L (its spectral radius) as small as possible. 
The whole point of overrelaxation is to discover this optimal @. The product of the 
eigenvalues equals det L = det T / det S: 


Aj Az = detL = (1- @)”. 


Always detS = detD because L lies below the diagonal, and det T = det(1 — @)D be- 
cause U lies above the diagonal. Their product is detL = (1 — @)". (This explains why 
we never go as far as @ = 2. The product of the eigenvalues would be too large, and 
the iteration could not converge.) We also get a clue to the behavior of the eigenvalues: 
At the optimal @ the two eigenvalues are equal. They must both equal @ —1 so their 
product will match detL. This value of @ is easy to compute, because the sum of the 
eigenvalues always agrees with the sum of the diagonal entries (the trace of L): 


1 
Optimal @ Ay + Az = (@opt — 1) + (@opt — 1) = 2 —2@opt + 7 Pope: (6) 
This quadratic equation gives Wopt = 4(2 — /3) & 1.07. The two equal eigenvalues are 
approximately @— 1 = 1.07, which is a major reduction from the Gauss-Seidel value 
A= t at @ = 1. In this example, the right choice of @ has again doubled the rate of 
convergence, because (4)? ~ .07. If @ is further increased, the eigenvalues become a 


complex conjugate pair—both have |A| = @ — 1, which is now increasing with @. 


The discovery that such an improvement could be produced so easily, almost as if by 
magic, was the starting point for 20 years of enormous activity in numerical analysis. 
The first problem was solved in Young’s 1950 thesis—a simple formula for the optimal 
@. The key step was to connect the eigenvalues A of L to the eigenvalues p of the 
original Jacobi matrix D~'(—L —U). That connection is expressed by 


Formula for @ (A+@—1)? =A@7p’. (7) 


This is valid for a wide class of finite difference matrices, and if we take w = 1 (Gauss- 
Seidel) it yields A7 = Au”. Therefore A = 0 and A = 1? as in Example 2, where ps = +3 
and A =0,A = i All the matrices in Young’s class have eigenvalues mw that occur in 
plus-minus pairs, and the corresponding A are 0 and p17. So Gauss-Seidel doubles the 


Jacobi rate of convergence. 


The important problem is to choose @ so that Amax will be minimized. Fortunately, 
Young’s equation (7) is exactly our 2 by 2 example! The best @ makes the two roots A 
both equal to @ — 1: 


2(1— V1—#?) 

Le 
For a large matrix, this pattern will be repeated for a number of different pairs +u;—and 
we can only make a single choice of @. The largest U gives the largest value of @ and 


(@—1)+(@—1)=2-20+p7o’, or O= 
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of A = m—1. Since our goal is to make Amax as small as possible, that extremal pair 
specifies the best choice opt: 


2(1 — Vv 1 = pe ae) 
Uinax 


7G_ The splittings of the —1, 2, —1 matrix of order n yield these eigenvalues 
of B: 


Optimal® = @pr = and = Amax = Wopt — 1. (8) 


Jacobi (S = 0, 2, 0 matrix): S-'T has |A|max = cos 4 
poe 
Gauss-Seidel (S = —1, 2, 0 matrix): SP has VA vie = (cos a 7 
nm \? nm \? 
SOR (with the best @): Alas = (cos = ) / (+5055) 


This can only be appreciated by an example. Suppose A is of order 21, which is very 
moderate. Then h = oan cosh = .99, and the Jacobi method is slow; cos* zh = .98 
means that even Gauss-Seidel will require a great many iterations. But since sinh = 
V.02 = .14, the optimal overrelaxation method will have the convergence factor 


86 
Amax = 1.14 


The error is reduced by 25% at every step, and a single SOR step is the equivalent of 
30 Jacobi steps: (.99)°° = .75. 

That is a striking result from such a simple idea. Its real applications are not in one- 
dimensional problems like —u,, = f. A tridiagonal system Ax = b is already easy. It is 
for partial differential equations that overrelaxation (and other ideas) will be important. 
Changing to —ux, — Uyy = f leads to the “five-point scheme.” The entries —1, 2, —1 in 
the x direction combine with —1, 2, —1 in the y direction to give a main diagonal of 
+4 and four off-diagonal entries of —1. The matrix A does not have a small bandwidth! 
There is no way to number the N* mesh points in a square so that each point stays 
close to all four of its neighbors. That is the true curse of dimensionality, and parallel 
computers will partly relieve it. 

If the ordering goes a row at a time, every point must wait a whole row for the neigh- 
bor above it to turn up. The “five-point matrix” has bandwidth N: This matrix has had 
more attention, and been attacked in more different ways, than any other linear equa- 
tion Ax = b. The trend now is back to direct methods, based on an idea of Golub and 
Hockney; certain special matrices will fall apart when they are dropped the right way. 
(It is comparable to the Fast Fourier Transform.) Before that came the iterative methods 
of alternating direction, in which the splitting separated the tridiagonal matrix in the x 
direction from the one in the y direction, A recent choice is $ = LoU o, in which small 


fey with opt = 1+ Amax = 1.75. 
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—1, 2, —lin x and y 
gives —1, —1, 4, -1, -1 


AS 


entries of the true L and U are set to zero while factoring A. It is called incomplete LU 
and it can be terrific. 

We cannot close without mentioning the conjugate gradient method, which looked 
dead hut is suddenly very much alive (Problem 33 gives the steps). It is direct rather 
than iterative, but unlike elimination, it can be stopped part way. And needless to say, 
a completely new idea may still appear and win. But it seems fair to say that it was the 
change from .99 to .75 that revolutionized the solution of Ax = b. 


Problem Set 7.4 


1. This matrix has eigenvalues 2 — /2, 2, and 2+ V2: 
2 —-1 O 


0 -1 2 
Find the Jacobi matrix D~'(—L—U) and the Gauss-Seidel matrix (D+ L)~!(—U) 
and their eigenvalues, and the numbers @pop and Amax for SOR. 


2. For this n by n matrix, describe the Jacobi matrix J = D~'(—L—U): 


2. - =) 


—l 
=1 2 


Show that the vector x; = (sinzh,sin27h,...,sinnzh) is an eigenvector of J with 
eigenvalue A; = costh = cosa/(n+ 1). 


3. In Problem 2, show that x, = (sinkzh, sin2kah,...,sinnk7h) is an eigenvector of A. 
Multiply x, by A to find the corresponding eigenvalue o;. Verify that in the 3 by 3 
case these eigenvalues are 2 — 2, 2,24 V2. 

Note. The eigenvalues of the Jacobi matrix J = }(—L—U) =1— 4A are Ay = 1— 
5 Ok = coskh. They occur in plus-minus pairs and Amax is cos 7h. 
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Problems 4—5 require Gershgorin’s “circle theorem”: Every eigenvalue of A lies in at 
least one of the circles C,,...,Cy, where C; has its center at the diagonal entry aj. Its 
radius rj = Viz; |a;;| is equal to the absolute sum along the rest of the row. 


Proof. Suppose x; is the largest component of x. Then Ax = Ax leads to 


10. 


. If A is an eigenvalue of A, then 


Xj 
(A — au)xi =o aijx;, or |A — aii < ¥ lal < Y laij| =7i:- 
i#i AP a 


. The matrix 


is called diagonally dominant because every |a;;| > rj. Show that zero cannot lie in 
any of the circles, and conclude that A is nonsingular. 


. Write the Jacobi matrix J for the diagonally dominant A of Problem 4, and find the 


three Gershgorin circles for J. Show that all the radii satisfy r; < 1, and that the 
Jacobi iteration converges. 


The true solution to Ax = b is slightly different from the elimination solution to 
LUx9 = b; A— LU misses zero because of roundoff. One strategy is to do everything 
in double precision, but a better and faster way is iterative refinement: Compute only 
one vector r = b — Axo in double precision, solve LU y = r, and add the correction y to 
Xo. Problem: Multiply x; = x9+ y by LU, write the result as a splitting Sx; = Txo +b, 
and explain why T is extremely small. This single step brings us almost exactly to x. 


a b 
c dl’ 
find the Jacobi iteration matrix S~'T = —D~'(L+U) and its eigenvalues p;. Find 


also the Gauss-Seidel matrix —(D+L)~'U and its eigenvalues A;, and decide whether 
Amax = eae 


For a general 2 by 2 matrix 


A= 


. Change Ax = b tox = (I—A)x+b. What are S and T for this splitting? What matrix 


S—'T controls the convergence of x;.; = (1—A)x, +b? 


is an eigenvalue of B = J —A. The real eigen- 
values of B have absolute value less than 1 if the real eigenvalues of A lie between 
and 


Show why the iteration x,4.; = (J —A)x, +b does not converge for A = [ *, 3']. 


11. 


12. 


13. 


14. 


15. 


16. 


17. 
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Why is the norm of B* never larger than ||B||‘? Then ||B|| < 1 guarantees that the 
powers Bk approach zero (convergence). This is no surprise, since |A |max is below 


|B. 


If A is singular, then all splittings A = S—T must fail. From Ax = 0, show that 
S~'Tx =x. So this matrix B= S~'T has A = 1 and fails. 


Change the 2s to 3s and find the eigenvalues of S~'T for both methods: 


3 0 0 1 
(J) : ; wi) 0 


Does |A|max for Gauss-Seidel equal |/ | 


xr +b (GS) 


2 


“vax Lor Jacobi? 


Write a computer code (MATLAB or other) for Gauss-Seidel. You can define S and 
T from A, or set up the iteration loop directly from the entries a;;. Test it on the —1, 
2, —1 matrices A of order 10, 20, 50, with b = (1,0,...,0). 


The SOR splitting matrix S is the same as for Gauss-Seidel except that the diagonal 
is divided by @. Write a program for SOR on an n by n matrix. Apply it with o = 1, 
1.4, 1.8, 2.2 when A is the —1, 2, —1 matrix of order 10. 


When A = A!, the Arnoldi-Lanczos method finds orthonormal q’s so that Aq j= 
bj—1qj-1 + ajqj+bjqj+11 (with go = 0). Multiply by qj to find a formula for a;. The 
equation says that AQ = QT where T isa matrix. 


What bound on |A|max does Gershgorin give For these matrices (see Problem 4)? 
What are the three Gershgorin circles that contain all the eigenvalues? 


The key point for large matrices is that matrix-vector multiplication is much 
faster than matrix-matrix multiplication. A crucial construction starts with a vec- 
tor b and computes Ab,A*b,... (but never A”!). The first N vectors span the Nth 
Krylov subspace. They are the columns of the Krylov matrix Ky: 


Ky =|b Ab A2b --- AN-1p]. 


The Arnoldi-Lanczos iteration orthogonalizes the columns of Ky, and the conjugate 
gradient iteration solves Ax = b when A is symmetric positive definite. 
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Arnoldi Iteration Conjugate Gradient Iteration 


qi =b/ >| x0 =0, ro = 5, po=ro 
forn=l1toN—1 forn=I1toN 
v =AGn On = (Fate) P) Apia) step length x,_1 to xp 
for j= 1ton Xn =Xn—-1 + OnPn-1 approximate solution 
n= qjv Tn =Tn—1 — On APn—1 new residual b — Ax,, 
v=v—hing; Br) aren) improvement this step 
hn+in = |\vI| Pn=TnatBnPn-1 next search direction 


Gn+1 = V/hn+i,n Note: Only 1 matrix vector multiplication Aq and Ap 


18. In Arnoldi, show that g2 is orthogonal to g,. The Arnoldi method is Gram-Schmidt 
orthogonalization applied to the Krylov matrix: Ky = QyRy. The eigenvalues of 
Ql AOn are often very close to those of A, even for N <n. The Lanczos iteration is 
Arnoldi for symmetric matrices (all coded in ARPACK). 


19. In conjugate gradients, show that r; is orthogonal to ro (orthogonal residuals), and 
p'Apo = 0 (search directions are A-orthogonal). The iteration solves Ax = b by 
minimizing the error e! Ae in the Krylov subspace. It is a fantastic algorithm. 


Linear Programming and Game Theory 


8.1 Linear Inequalities 


Algebra is about equations, and analysis is often about inequalities. The line between 
them has always seemed clear. But I have realized that this chapter is a counterexam- 
ple: linear programming is about inequalities, but it is unquestionably a part of linear 
algebra. It is also extremely useful—business decisions are more likely to involve linear 
programming than determinants or eigenvalues. 

There are three ways to approach the underlying mathematics: intuitively through 
the geometry, computationally through the simplex method, or algebraically through 
duality. These approaches are developed in Sections 8.1, 8.2, and 8.3. Then Section 
8.4 is about problems (like marriage) in which the solution is an integer. Section 8.5 
discusses poker and other matrix games. The MIT students in Bringing Down the House 
counted high cards to win at blackjack (Las Vegas follows fixed rules, and a true matrix 
game involves random strategies). 

Section 8.3 has something new in this fourth edition. The simplex method is now 
in a lively competition with a completely different way to do the computations, called 
an interior point method. The excitement began when Karmarkar claimed that his 
version was 50 times faster than the simplex method. (His algorithm, outlined in 8.2, 
was one of the first to be patented—something we then believed impossible, and not 
really desirable.) That claim brought a burst of research into methods that approach 
the solution from the “interior” where all inequalities are strict: x > 0 becomes x > 0. 
The result is now a great way to get help from the dual problem in solving the primal 
problem. 

One key to this chapter is to see the geometric meaning of linear inequalities. An 
inequality divides n-dimensional space into a halfspace in which the inequality is satis- 
fied, and a halfspace in which it is not. A typical example is x + 2y > 4. The boundary 
between the two halfspaces is the line x-++ 2y = 4, where the inequality is “tight.” Figure 
8.1 would look almost the same in three dimensions. The boundary becomes a plane 
like x+ 2y+z=4, and above it is the halfspace x ++ 2y+z > 4. In n dimensions, the 
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“plane” has dimension n — 1. 


Figure 8.1: Equations give lines and planes. Inequalities give halfspaces. 


Another constraint is fundamental to linear programming: x and y are required to be 
nonnegative. This pair of inequalities x > 0 and y > 0 produces two more halfspaces. 
Figure 8.2 is bounded by the coordinate axes: x > 0 admits all points to the right of 
x =0, and y > 01s the halfspace above y = 0. 


The Feasible Set and the Cost Function 


The important step is to impose all three inequalities at once. They combine to give the 
shaded region in Figure 8.2. This feasible set is the intersection of the three halfspaces 
x+2y >4,x > 0, and y > 0. A feasible set is composed of the solutions to a family of 
linear inequalities like Ax > b (the intersection of m halfspaces). When we also require 
that every component of x is nonnegative (the vector inequality x > 0), this adds n more 
halfspaces. The more constraints we impose, the smaller the feasible set. 

It can easily happen that a feasible set is bounded or even empty. If we switch our 
example to the halfspace x + 2y < 4, keeping x > 0 and y > 0, we get the small triangle 
OAB. By combining both inequalities x + 2y > 4 and x+ 2y < 4, the set shrinks to a line 
where x + 2y = 4. If we add a contradictory constraint like x + 2y < —2, the feasible set 
is empty. 

The algebra of linear inequalities (or feasible sets) is one part of our subject. But 
linear programming has another essential ingredient: It looks for the feasible point that 
maximizes or minimizes a certain cost function like 2x + 3y. The problem in linear 
programming is to find the point that lies in the feasible set and minimizes the cost. 

The problem is illustrated by the geometry of Figure 8,2. The family of costs 2x + 3y 
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gives a family of parallel lines. The minimum cost comes when the first line intersects 
the feasible set. That intersection occurs at B, where x* = 0 and y* = 2; the minimum 
cost is 2x* + 3y* = 6. The vector (0,2) is feasible because it lies in the feasible set, it is 
optimal because it minimizes the cost function, and the minimum cost 6 is the value of 
the program. We denote optimal vectors by an asterisk. 


i] 
cost 2x2 + 38y = 6 feasible set 
ji es E+2y>4 
2e+3y=0 “i z>0 


Figure 8.2: The feasible set with flat sides, and the costs 2x + 3y, touching at B. 


The optimal vector occurs at a corner of the feasible set. This is guaranteed by the 
geometry, because the lines that give the cost function (or the planes, when we get to 
more unknowns) move steadily up until they intersect the feasible set. The first contact 
must occur along its boundary! The “simplex method” will go from one corner of the 
feasible set to the next until it finds the corner with lowest cost. In contrast, “interior 
point methods” approach that optimal solution from inside the feasible set. 


Note. With a different cost function, the intersection might not be just a single point. If 
the cost happened to be x + 2y, the whole edge between B and A would be optimal. The 
minimum cost is x* + 2y*, which equals 4 for all these optimal vectors. On our feasible 
set, the maximum problem would have no solution! The cost can go arbitrarily high and 
the maximum cost is infinite. 


Every linear programming problem falls into one of three possible categories: 
1. The feasible set is empty. 
2. The cost function is unbounded on the feasible set. 
3. The cost reaches its minimum (or maximum) on the feasible set: the good case. 


The empty and unbounded cases should be very uncommon for a genuine problem in 
economics or engineering. We expect a solution. 
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Slack Variables 


There is a simple way to change the inequality x + 2y > 4 to an equation. Just introduce 
the difference as a slack variable w = x+2y—4. This is our equation! The old con- 
straint x + 2y > 4 is converted into w > 0, which matches perfectly the other inequality 
constraints x > 0, y > 0. Then we have only equations and simple nonnegativity con- 
straints on x, y, w. The variables w that “take up the slack” are now included in the vector 
unknown x: 


Primal problem Minimize cx subject to Ax = b and x > 0. 


The row vector c contains the costs; in our example, c = [2 3 0]. The condition x > 0 
puts the problem into the nonnegative part of R”. Those inequalities cut back on the 
solutions to Ax = b. Elimination is in danger, and a completely new idea is needed. 


The Diet Problem and Its Dual 


Our example with cost 2x + 3y can be put into words. It illustrates the “diet problem” 
in linear programming, with two sources of protein—say steak and peanut butter. Each 
pound of peanut butter gives a unit of protein, and each steak gives two units. At least 
four units are required in the diet. Therefore a diet containing x pounds of peanut butter 
and y steaks is constrained by x + 2y > 4, as well as by x > 0 and y > 0. (We cannot have 
negative steak or peanut butter.) This is the feasible set, and me p1001cm is to minimize 
the cost. If a pound of peanut butter costs $2 and a steak is $3. then the cost of the whole 
diet is 2x + 3y. Fortunately, the optimal diet is two steaks: x* = 0 and y* = 2. 

Every linear program, including this one, has a dual. If the original prohelv a min- 
imization, its dual is a maximization. The minimum in the given “primal problem” 
equals the maximum in its dual. This is the key to linear programming, and it will be 
explained in Section 8.3. Here we stay with the diet problem and try to interpret its dual. 

In place of the shopper, who buys enough protein at minimal cost, the dual problem 
is faced by a druggist. Protein pills compete with steak and peanut butter. Immediately 
we meet the two ingredients of a typical linear program: The druggist maximizes the 
pill price p, but that price is subject to linear constraints. Synthetic protein must not 
cost more than the protein in peanut butter ($2 a unit) or the protein in steak ($3 for two 
units). The price must be nonnegative or the druggist will not sell. Since four units of 
protein are required, the income to the druggist will be 4p: 


Dual problem Maximize 4p, subject to p < 2, 2p < 3, and p > 0. 


In this example the dual is easier to solve than the primal; it has only one unknown 
p. The constraint 2p < 3 is the tight one that is really active, and the maximum price 
of synthetic protein is p = $1.50. The maximum revenue is 4p = $6, and the shopper 
ends up paying the same for natural and synthetic protein. That is the duality theorem: 
maximum equals minimum. 
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Typical Applications 


The next section will concentrate on solving linear programs. This is the time to describe 
two practical situations in which we minimize or maximize a linear cost function subject 
to linear constraints. 


1. Production Planning. Suppose General Motors makes a profit of $200 on each 
Chevrolet, $300 on each Buick, and $500 on each Cadillac. These get 20, 17, and 14 
miles per gallon, respectively, and Congress insists that the average car must get 18. The 
plant can assemble a Chevrolet in 1 minute, a Buick in 2 minutes, and a Cadillac in 3 
minutes. What is the maximum profit in 8 hours (480 minutes)? 


Problem Maximize the profit 200x + 300y + 500z subject to 


20x + 17y+ 14z > 18(*#+y+z), x+2y+3z < 480, bin eae US 


2. Portfolio Selection. Federal bonds pay 5%, municipals pay 6%, and junk bonds pay 
9%. We can buy amounts x, y, z not exceeding a total of $100,000. The problem is to 
maximize the interest, with two constraints: 


(i) no more than $20,000 can be invested in junk bonds, and 


(ii) the portfolio’s average quality must be no lower than municipals, so x > z. 


Problem Maximize 5x + 6y+ 9z subject to 
x+y+z< 100,000, z < 20,000, ae x,y,z = 0. 


The three inequalities give three slack variables, with new equations like w = x — z and 
inequalities w > 0. 


Problem Set 8.1 


1. Sketch the feasible set with constraints x + 2y > 6, 2x+y > 6, x >0, y>0. What 
points lie at the three “corners” of this set? 


2. (Recommended) On the preceding feasible set, what is the minimum value of the 
cost function x+y? Draw the line x + y = constant that first touches the feasible set. 
What points minimize the cost functions 3x + y and x — y? 


3. Show that the feasible set constrained by 2x +5y < 3, —3x+8y < —5,x >0, y> 0, 
is empty. 
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4. Show that the following problem is feasible but unbounded, so it has no optimal 
solution: Maximize x+ y, subject tox > 0, y > 0, —3x+2y< —-l,x-—y <2. 


5. Add a single inequality constraint to x > 0, y > 0 such that the feasible set contains 
only one point. 


6. What shape is the feasible set x > 0, y > 0, z > 0, x+y+z= 1, and what is the 
maximum of x+ 2y+ 3z? 


7. Solve the portfolio problem at the end of the preceding section. 


8. In the feasible set for the General Motors problem, the nonnegativity x, y,z > 0 leaves 
an eighth of three-dimensional space (the positive octant). How is this cut by the two 
planes from the constraints, and what shape is the feasible set? How do its corners 
show that, with only these two constraints, there will be only two kinds of cars in the 
optimal solution? 


9. (Transportation problem) Suppose Texas, California, and Alaska each produce a mil- 
lion barrels of oil; 800,000 barrels are needed in Chicago at a distance of 1000, 2000, 
and 3000 miles from the three producers, respectively; and 2,200,000 barrels are 
needed in New England 1500, 3000, and 3700 miles away. If shipments cost one 
unit for each barrel-mile, what linear program with five equality constraints must be 
solved to minimize the shipping cost? 


8.2 The Simplex Method 


This section is about linear programming with n unknowns x > 0 and m constraints 
Ax = b. In the previous section we had two variables, and one constraint x + 2y > 4. The 
full problem is not hard to explain, and not easy to solve. 

The best approach is to put the problem into matrix form. We are given A, b, and c: 


1. anm by n matrix A. 
2. acolumn vector b with m components, and 
3. arow vector c (cost vector) with n components. 


To be “feasible,” the vector x must satisfy x > 0 and Ax > b. The optimal vector x* is the 
feasible vector of least cost—and the cost 1s cx = C1xX1 +++: + CynXp. 


Minimum problem Minimize the cost cx, subject to x > 0 and Ax > b. 


The condition x > 0 restricts x to the positive quadrant in n-dimensional space. In 
R’ it is a quarter of the plane; it is an eighth of R°. A random vector has one chance 
in 2” of being nonnegative. Ax > b produces m additional halfspaces, and the feasible 
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vectors meet all of the m+n conditions. In other words, x lies in the intersection of 
m-+n halfspaces. This feasible set has flat sides; it may be unbounded. and it may be 
empty. 

The cost function cx brings to the problem a family of parallel planes. One plane 
cx = 0 goes through the origin. The planes cx = constant give all possible costs. As 
the cost varies, these planes sweep out the whole n-dimensional space. The optimal x* 
(lowest cost) occurs at the point where the planes first touch the feasible set. 

Our aim is to compute x*. We could do it (in principle) by finding all the corners 
of the feasible set, and computing their costs. In practice this is impossible. There 
could be billions of corners, and we cannot compute them all. Instead we turn to the 
simplex method, one of the most celebrated ideas in computational mathematics. It was 
developed by Dantzig as a systematic way to solve linear programs, and either by luck 
or genius it is an astonishing success. The steps of the simplex method are summarized 
later, and first we try to explain them. 


The Geometry: Movement Along Edges 


I think it is the geometric explanation that gives the method away. Phase I simply locates 
one corner of the feasible set. The heart of the method goes from corner to corner 
along the edges of the feasible set. At a typical corner there are n edges to choose from. 
Some edges lead away from the optimal but unknown x*, and others lead gradually 
toward it. Dantzig chose an edge that leads to a new corner with a lower cost. There 
is no possibility of returning to anything more expensive. Eventually a special corner is 
reached, from which all edges go the wrong way: The cost has been minimized. That 
corner is the optimal vector x*, and the method stops. 

The next problem is to turn the ideas of corner and edge into linear algebra. A corner 
is the meeting point of n different planes. Each plane is given by one equation—just 
as three planes (front wall, side wall, and floor) produce a corner in three dimensions. 
Each corner of the feasible set comes from turning n of the n +m inequalities Ax > b 
and x > 0 into equations, and finding the intersection of these n planes. 

One possibility is to choose the n equations x; = 0,...,x, = 0, and end up at the 
origin. Like all the other possible choices, this intersection point will only be a genuine 
corner if it also satisfies the m remaining inequality constraints. Otherwise it is not even 
in the feasible set, and is a complete fake. Our example with n = 2 variables and m = 2 
constraints has six intersections, illustrated in Figure 8.3. Three of them are actually 
corners P, Q, R of the feasible set. They are the vectors (0,6), (2,2), and (6,0), One 
of them must be the optimal vector (unless the minimum cost is —°o). The other three, 
including the origin, are fakes. 

In general there are (n+ m)!/n!m! possible intersections. That counts the number of 
ways to choose n plane equations out of n+ m. The size of that binomial coefficient 
makes computing all corners totally impractical for large m and n. It is the task of Phase 


424 Chapter 8 Linear Programming and Game Theory 


feasible set 


Figure 8.3: The corners P, Q, R, and the edges of the feasible set. 


I either to find one genuine corner or to establish that the feasible set is empty. We 
continue on the assumption that a corner has been found. 

Suppose one of the n intersecting planes is removed. The points that satisfy the 
remaining n— | equations form an edge that comes out of the corner. This edge 
is the intersection of the n — 1 planes. To stay in the feasible set, only one direction is 
allowed along each edge. But we do have a choice of n different edges, and Phase II 
must make that choice. 

To describe this phase, rewrite Ax > b in a form completely parallel to the n simple 
constraints x; > 0. This is the role of the slack variables w = Ax — b. The constraints 
Ax > b are translated into w; > 0,...,Wm => 0, with one slack variable for every row of 
A. The equation w = Ax — b, or Ax — w = b, goes into matrix form: 


Slack variables give m equations lA =] | =p; 
w 


The feasible set is governed by these m equations and the n+m simple inequalities x > 0, 
w > 0. We now have equality constraints and nonnegativity. 
The simplex method notices no difference between x and w, so we simplify: 


lA -1] is renamed A | is renamed x E 0| is renamed c. 
w 


The equality constraints are now Ax = b. The n+ ™m inequalities become just x > 0. 
The only trace left of the slack variable w is in the fact that the new matrix A is m by 
n-+m, and the new x has n+ m components. We keep this much of the original notation 
leaving m and n unchanged as a reminder of what happened. The problem has become: 
Minimize cx, subject to x > 0 and Ax = b. 
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Example 1. The problem in Figure 8.3 has constraints x + 2y > 6, 2x+y > 6, and cost 
x+y. The new system has four unknowns (x, y, and two slack variables): 


A=) : m “| (9 c=[1 1 0 of. 


The Simplex Algorithm 


With equality constraints, the simplex method can begin. A corner is now a point where 
n components of the new vector x (the old x and w) are zero. These n components of x 
are the free variables in Ax = b. The remaining m components are the basic variables or 
pivot variables. Setting the n free variables to zero, the m equations Ax = b determine 
the m basic variables. This “basic solution” x will be a genuine corner if its m nonzero 
components are positive. Then x belongs to the feasible set. 


8A The corners of the feasible set are the basic feasible solutions of Ax = b. 
A solution is basic when n of its m+n components are zero, and it is feasible 
when it satisfies x > 0. Phase I of the simplex method finds one basic feasible 
solution. Phase II moves step by step to the optimal x*. 


The corner point P in Figure 8.3 is the intersection of x = 0 with 2x + y—6=0. 


Corner (0,6,6,0) 
Basic (two zeros) Ax = 


Feasible (positive nonzeros) 


Which corner do we go to next? We want to move along an edge to an adjacent 
corner. Since the two corners are neighbors, m— 1 basic variables will remain basic. 
Only one of the 6s will become free (zero). At the same time, one variable will move up 
from zero to become basic. The other m— 1 basic components (in this case, the other 6) 
will change but stay positive. The choice of edge (see Example 2 below) decides which 
variable leaves the basis and which one enters. The basic variables are computed by 
solving Ax = b. The free components of x are set to zero. 


Example 2. An entering variable and a leaving variable move us to a new corner. 
xy +x3+6x4+2x5=8 


Minimize 7x3—x4—3x5 subject to 
x2 + x3 +3x5=9. 


Start from the corner at which x; = 8 and x2 = 9 are the basic variables. At that corner 
x3 =X4 =x5 =0. This is feasible, but the zero cost may not be minimal. It would 
be foolish to make x3 positive, because its cost coefficient is +7 and we are trying to 
lower the cost. We choose x5 because it has the most negative cost coefficient —3. The 
entering variable will be xs. 
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With x5 entering the basis, x; or x2 must leave. In the first equation, increase x5 and 
decrease x; while keeping x; + 2x5 = 8. Then x; will be down to zero when x5 reaches 
4. The second equation keeps x2 + 3x5 = 9. Here x5 can only increase as far as 3. To 
go further would make x2 negative, so the leaving variable is x2. The new corner has 
x = (2,0,0,0,3). The cost is down to —9. 


Quick Way In Ax = dD, the right sides divided by the coefficients of the entering 
variable are : and 2, The smallest ratio 3 tells which variable hits zero first, and must 
leave. We consider only positive ratios, because if the coefficient of x5 were —3, then 
increasing x5 would actually increase x2. (At x5 = 10 the second equation would give 
x2 = 39.) The ratio : says that the second variable leaves. It also gives x5 = 3. 

If all coefficients of x5 had been negative, this would be an unbounded case: we can 
make x5 arbitrarily large, and bring the cost down toward —c9, 

The current step ends at the new corner x = (2,0,0,0,3). The next step will only be 
easy if the basic variables x; and x5 stand by themselves (as x; and x2 originally did). 
Therefore, we “pivot” by substituting x5 = 3(9 — x2 — x3) into the cost function and the 
first equation. The new problem, starting from the new corner, is: 


Minimize the cost 7x3 —X4— (9 —x2 — x3) = x0 +843 —214-9 


: ; x1 — 3x2 + 4x3 + 6x4 = 2, 

with constraints i 1 
3X2 + 3X3 +x5=3. 

The next step is now easy. The only negative coefficient —1 in the cost makes x4 the 

entering variable. The ratios of z and 3, the right sides divided by the x4 column, make 

x, the leaving variable. The new corner is x* = (0,0,0, i, ). The new cost —94 is the 

minimum. 

In a large problem, a departing variable might reenter the basis later on. But the cost 
keeps going down—except in a degenerate case—so the m basic variables can’t be the 
same as before. No corner is ever revisited! The simplex method must end at the optimal 
corner (or at —co if the cost turns out to be unbounded). What is remarkable is the speed 


at which x* is found. 


Summary The cost coefficients 7, —1, —3 at the first corner and 1, 8, —1 at the 
second corner decided the entering variables. (These numbers go into r, the crucial vec- 
tor defined below. When they are all positive we stop.) The ratios decided the leaving 
variables. 


Remark on Degeneracy A corner is degenerate if more than the usual n com- 
ponents of x are zero. More than n planes pass through the corner, so a basic variable 
happens to vanish. The ratios that determine the leaving variable will include zeros, and 
the basis might change without actually moving from the corner. In theory, we could 
stay at a corner and cycle forever in the choice of basis. 
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Fortunately, cycling does not occur. It is so rare that commercial codes ignore it. 
Unfortunately, degeneracy is extremely common in applications—if you print the cost 
after each simplex step you see it repeat several times before the simplex method finds a 
good edge. Then the cost decreases again. 


The Tableau 


Each simplex step involves decisions followed by row operations—the entering and 
leaving variables have to be chosen, and they have to be made to come and go. One 
way to organize the step is to fit A, b, c into a large matrix, or tableau: 


A b 


Tableau is m+ 1 by m+n+1 = ol: 


At the start, the basic variables may be mixed with the free variables. Renumbering if 
necessary, suppose that x1,...,Xm are the basic (nonzero) variables at the current corner. 
The first m columns of A form a square matrix B (the basis matrix for that corner). The 
last n columns give an m by n matrix N. The cost vector c splits into [cg cy], and the 
unknown x into (xg,xy). 

At the corner, the free variables are xy = 0. There, Ax = b turns into Bxg = b: 


Tableau at corner T= Faas ae | eH. ApS B'b cost= cpB'b. 
The basic variables will stand alone when elimination multiplies by B7!: 


Reduced tableau T= 


To reach the fully reduced row echelon form R = rref(T), subtract cg times the top 
block row from the bottom row: 


Fully reduced = 


A BING BOB 
0 icen —cpB'!N ‘—cpB 'b 


Let me review the meaning of each entry in this tableau, and also call attention to Ex- 
ample 3 (following, with numbers). Here is the algebra: 


Constraints xz +B7'!Nxy =B'b Corner x3 = Bo'b, xn = 0. (1) 
The cost cgxgp + cnyxn has been turned into 
Cost cx = (cy — cpB'N)xy +cpBo!b Cost at this corner = cpB~!b. (2) 


Every important quantity appears in the fully reduced tableau R. We can decide whether 
the corner is optimal by looking at r = cy — cgB™!N in the middle of the bottom row. If 
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any entry in r is negative, the cost can still be reduced. We can make rxy negative, 
at the start of equation (2), by increasing a component of xy. That will be our next step. 
But if r > 0, the best corner has been found. This is the stopping test, or optimality 
condition: 


8B The corner is optimal when r = cy — cpB-!N > 0. Its cost is cgB~'b. 
Negative components of r correspond to edges on which the cost goes down. 
The entering variable x; corresponds to the most negative component of r. 


The components of r are the reduced costs—the cost in cy to use a free variable 
minus what it saves. Computing r is called pricing out the variables. If the direct cost 
(in cy) is less than the saving (from reducing basic variables), then r; < 0, and it will pay 
to increase that free variable. 

Suppose the most negative reduced cost is r;. Then the ith component of xy is the 
entering variable, which increases from zero to a positive value @ at the next corner 
(the end of the edge). 

As x; is increased, other components of x may decrease (to maintain Ax = b). The x; 
that reaches zero first becomes the leaving variable—it changes from basic to free. We 
reach the next corner when a component of xg drops to zero. 

That new corner is feasible because we still have x > 0. It is basic because we again 
have n zero components. The ith component of xy went from zero to a. The kth com- 
ponent of xg dropped to zero (the other components of xg remain positive). The leaving 
x; that drops to zero is the one that gives the minimum ratio in equation (3): 


8C Suppose x; is the entering variable and u is column i of N: 


B A 
At new corner x; = @ = smallest ratio (B- = ; (3) 

ys 
This minimum is taken only over positive components of B~!u. The kth col- 
umn of the old B leaves the basis (x; becomes 0) and the new column vu enters. 


B~'uw is the column of B~'N in the reduced tableau R, above the most negative entry in 
the bottom row r, If B~'w < 0, the next corner is infinitely far away and the minimal 
cost is —co (this doesn’t happen here). Our example will go from the corner P to Q, and 
begin again at Q. 


Example 3. The original cost function x + y and constraints Ax = b = (6,6) give 


At the corner P in Figure 8.3, x = 0 intersects 2x ++ y = 6. To be organized, we exchange 
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columns | and 3 to put basic variables before free variables: 


Tableau at P 2 a 


Then, elimination multiplies the first row by —1, to give a unit pivot, and uses the second 
row to produce zeros in the second column: 


1 
Fully reduced at P R=] 0 


Look first at r = {—1 1] in the bottom row. It has a negative entry in column 3, so the 
third variable will enter the basis. The current corner P and its cost +6 are not optimal. 
The column above that negative entry is B~'u = (3,2); its ratios with the last column 
are 4 and g. Since the first ratio is smaller, the first unknown w (and the first column of 
the tableau) is pushed out of the basis. We move along the feasible set from corner P to 
corner Q in Figure 8.3. 

The new tableau exchanges columns | and 3, and pivoting by elimination gives 


1 Ds 
Be, ae. 
3 0:1 -2:6 a oe 
oe ad ee ee oe 
-1 0,0 1 :-6 0 0:4 41-4 


In that new tableau at Q, r = [3 5] is positive. The stopping test is passed. The corner 
x = y =2 and its cost +4 are optimal. 


The Organization of a Simplex Step 


The geometry of the simplex method is now expressed in algebra—“corners” are “basic 
feasible solutions.” The vector r and the ratio @ are decisive. Their calculation is the 
heart of the simplex method, and it can be organized in three different ways: 


1. In a tableau, as above. 
2. By updating B~! when column u taken from N replaces column k of B. 
3. By computing B = LU, and updating these LU factors instead of B7!. 


This list is really a brief history of the simplex method, In some ways, the most 
fascinating stage was the first—the tableau—which dominated the subject for so many 
years. For most of us it brought an aura of mystery to linear programming, chiefly 
because it managed to avoid matrix notation almost completely (by the skillful device of 
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writing out all matrices in full !). For computational purposes (except for small problems 
in textbooks), the day of the tableau is over. 

To see why, remember that after the most negative coefficient in r indicates which 
column u will enter the basis, none of the other columns above r will be used. Jt was 
a waste of time to compute them. In a larger problem, hundreds of columns would be 
computed time and time again, just waiting for their turn to enter the basis. It makes 
the theory clear to do the eliminations so completely and reach R. But in practice this 
cannot be justified. 

It is quicker, and in the end simpler, to see what calculations are really necessary. 
Each simplex step exchanges a column of N for a column of B. Those columns are 
decided by r and a. This step begins with the current basis matrix B and the current 
solution xg = Bo !b. 


A Step of the Simplex Method 


1. Compute the row vector A = cpB™! and the reduced costs r = cy — AN. 


2. If r > 0, stop: the current solution is optimal. Otherwise, if 7; is the most 
negative component, choose u = column i of N to enter the basis. 

3. Compute the ratios of B~'b to B~!u, admitting only positive components 
of B~'u. (If B~!u < 0, the minimal cost is —co.) When the smallest ratio 
occurs at component k, the kth column of the current B will leave. 


4. Update B, B™!, or LU, and the solution xg = B~'b. Return to step 1. 


This is sometimes called the revised simplex method to distinguish it from the oper- 
ations on a tableau. It is really the simplex method itself, boiled down. 
This discussion is finished once we decide how to compute steps 1, 3, and 4: 


A =cpBl," v=B lu, and xp=B'b. (4) 


The most popular way is to work directly with B~!, calculating it explicitly at the first 
corner. At succeeding corners, the pivoting step is simple. When column k of the identity 
matrix is replaced by u, column k of B~! is replaced by v = B~!w. To recover the identity 
matrix, elimination will multiply the old B~! by 

= 


1 V1 1 —v1/VE 
E'= VE = 1/v_ (5) 


Many simplex codes use the product form of the inverse, which saves these simple 
matrices E~! instead of directly updating B~'!. When needed, they are applied to b and 
cg. At regular intervals (maybe every 40 simplex steps), B~! is recomputed and the E~! 
are erased. Equation (5) is checked in Problem 9 at the end of this section. 
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A newer approach uses the ordinary methods of numerical linear algebra, regarding 
equation (4) as three equations sharing the same matrix B: 


AB=cp, Bv=u, Bxp = b. (6) 


The usual factorization B = LU (or PB = LU, with row exchanges for stability) leads to 
the three solutions. L and U can be updated instead of recomputed. 

One question remains: How many simplex steps do we have to take? This is impos- 
sible to answer in advance. Experience shows that the method touches only about 3m/2 
different corners, which means an operation count of about mn. That is comparable to 
ordinary elimination for Ax = b, and is the reason for the simplex method’s success. But 
mathematics shows that the path length cannot always be bounded by any fixed multiple 
or power of m. The worst feasible sets (Klee and Minty invented a lopsided cube) can 
force the simplex method to try every corner—at exponential cost. 

It was Khachian’s method that showed that linear programming could be solved in 
polynomial time.! His algorithm stayed inside the feasible set, and captured x* in a series 
of shrinking ellipsoids. Linear programming is in the nice class P, not in the dreaded 
class NP (like the traveling salesman problem). For VP problems it is believed (but not 
proved) that all deterministic algorithms must take exponentially long to finish, in the 
worst case. 

All this time, the simplex method was doing the job—in an average time that is now 
proved (for variants of the usual method) to be polynomial. For some reason, hidden in 
the geometry of many-dimensional polyhedra, bad feasible sets are rare and the simplex 
method is lucky. 


Karmarkar’s Method 


We come now to the most sensational event in the recent history of linear programming. 
Karmarkar proposed a method based on two simple ideas, and in his experiments it 
defeated the simplex method. The choice of problem and the details of the code are both 
crucial, and the debate is still going on. But Karmarkar’s ideas were so natural, and fit 
so perfectly into the framework of applied linear algebra, that they can be explained in a 
few paragraphs. 

The first idea is to start from a point inside the feasible set—we will suppose it is 
xo = (1,1,...,1). Since the cost is cx, the best cost-reducing direction is toward —c. 
Normally that takes us off the feasible set; moving in that direction does not maintain 
Ax = b. If Ax® = b and Ax! = b, then Ax = x! — x° has to satisfy AAx = 0. The step Ax 
must lie in the nullspace of A. Therefore we project —c onto the nullspace, to find the 
feasible direction closest to the best direction. This is the natural but expensive step in 


Karmarkar’s method. 

'The number of operations is bounded by powers of m and n, as in elimination. For integer programming and 
factoring into primes, all known algorithms can take exponentially long. The celebrated conjecture “P 4 NP” says 
that such problems cannot have polynomial algorithms. 
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The step Ax is a multiple of the projection —Pc. The longer the step, the more the 
cost is reduced—but we cannot go out of the feasible set. The multiple of —Pc is chosen 
so that x! is close to, but a little inside, the boundary at which a component of x reaches 
zero. 

That completes the first idea—the projection that gives the steepest feasible descent. 
The second step needs a new idea. since to continue in the same direction is useless. 

Karmarkar’s suggestion is to transform x! back to (1,1,...,1) at the center. His 
change of variables was nonlinear, but the simplest transformation is just a rescaling by 
a diagonal matrix D. Then we have room to move. The rescaling from x to X = D~!x 
changes the constraint and the cost: 


Ax=b becomes ADX =b c'x becomes c!DX. 


Therefore the matrix AD takes the place of A, and the vector c'D takes the place of c’. 
The second step projects the new c onto the nullspace of the new A. All the work is in 
this projection, to solve the weighted normal equations: 


(AD*A')y = AD’c. (7) 


The normal way to compute y is by elimination. Gram-Schmidt will orthogonalize the 
columns of DA’, which can be expensive (although it makes the rest of the calculation 
easy). The favorite for large sparse problems is the conjugate gradient method, which 
gives the exact answer y more slowly than elimination, but you can go part way and then 
stop. In the middle of elimination you cannot stop. 

Like other new ideas in scientific computing, Karmarkar’s method succeeded on some 
problems and not on others. The underlying idea was analyzed and improved. Newer 
interior point methods (staying inside the feasible set) are a major success—mentioned 
in the next section. And the simplex method remains tremendously valuable. like the 
whole subject of linear programming—which was discovered centuries after Ax = b, but 
shares the fundamental ideas of linear algebra. The most far-reaching of those ideas is 
duality, which comes next. 


Problem Set 8.2 


1. Minimize x; + x2 —x3, subject to 
2x1 — 4x0 +%3+%4 =4 
3x, + 5x2 +3 +x5=2. 


Which of x1, x2, x3 should enter the basis, and which of x4, x5 should leave? Compute 
the new pair of basic variables, and find the cost at the new corner. 


2. After the preceding simplex step, prepare for and decide on the next step. 


10. 


11. 


12. 
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In Example 3, suppose the cost is 3x+ y. With rearrangement, the cost vector is 
c = (0,1,3,0). Show that r > 0 and, therefore, that corner P is optimal. 


Suppose the cost function in Example 3 is x — y, so that after rearrangement c = 
(0, —1,1,0) at the corner P. Compute r and decide which column u should enter the 
basis. Then compute B~!u and show from its sign that you will never meet another 
corner. We are climbing the y-axis in Figure 8.3, and x — y goes to —°9., 


Again in Example 3, change the cost to x + 3y. Verify that the simplex method takes 
you from P to Q to R, and that the corner R is optimal. 


Phase I finds a basic feasible solution to Ax = b (a corner). After changing signs 
to make b > 0, consider the auxiliary problem of minimizing w; + w2+---+Wm», 
subject to x > 0, w > 0, Ax-+w = b. Whenever Ax = b has a nonnegative solution, 
the minimum cost in this problem will be zero—with w* = 0. 


(a) Show that, for this new problem, the corner x = 0, w = b is both basic and fea- 
sible. Therefore its Phase I is already set, and the simplex method can proceed 
to find the optimal pair x*, w*. If w* = 0, then x* is the required corner in the 
original problem. 

(b) With A = [1 1] and b = [3], write out the auxiliary problem, its Phase I vector 
x =0, w =), and its optimal vector. Find the corner of the feasible set x; — x2 = 3, 
xX, > x2 > 0, and draw a picture of this set. 


If we wanted to maximize instead of minimize the cost (with Ax = b and x > 0), 
what would be the stopping test on 7, and what rules would choose the column of NV 
to make basic and the column of B to make free? 


Minimize 2x; + x2, subject to xy +x2 > 4, x1 +3x2 > 12, x1 —x2 > 0, x > 0. 


Verify the inverse in equation (5), and show that BE has By = u in its kth column. 
Then BE is the correct basis matrix for the next stop, E~!B! is its inverse, and E~! 


updates the basis matrix correctly. 
Suppose we want to minimize cx = x; — x2, subject to 

2x; —4x2 +.x3 =6 
(all x1 ,x2,x3,x4 > 0). 
3x1 + 6x2 +x4= 12 


Starting from x = (0,0,6, 12), should x; or x2 be increased from its current value of 
zero? How far can it be increased until the equations force x3 or x4 down to zero? At 
that point, what is the new x? 


For the matrix P = ]— A'(AA')~'A, show that if x is in the nullspace of A, then 
Px =x. The nullspace stays unchanged under this projection. 


(a) Minimize the cost cx = 5x; + 4x7 + 8x3 on the plane x; + x2 +.x3 = 3, by testing 
the vertices P, Q, R, where the triangle is cut off by the requirement x > 0. 
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(b) Project c = (5,4,8) onto the nullspace of A= [1 1 1], and find the maximum 
step s that keeps e — sPc nonnegative. 


8.3 The Dual Problem 


Elimination can solve Ax = b, but the four fundamental subspaces showed that a different 
and deeper understanding is possible. It is exactly the same for linear programming. The 
mechanics of the simplex method will solve a linear program, but duality is really at the 
center of the underlying theory. Introducing the dual problem is an elegant idea, and 
at the same time fundamental for the applications. We shall explain as much as we 
understand. 

The theory begins with the given primal problem: 


Primal (P) Minimize cx, subject to x > 0 and Ax => b. 


The dual problem starts from the same A, b, and c, and reverses everything. In the 
primal, c is in the cost function and b is in the constraint, In the dual, b and c are 
switched, The dual unknown y is a row vector with m components, and the feasible set 
has yA < c instead of Ax > b. 

In short, the dual of a minimum problem is a maximum problem. Now y > 0: 


Dual (D) Maximize yb, subject to y > 0Oand yA <c. 


The dual of this problem is the original minimum problem. There is complete symmetry 
between the primal and dual problems. The simplex method applies equally well to a 
maximization—anyway, both problems get solved at once. 

I have to give you some interpretation of all these reversals. They conceal a competi- 
tion between the minimizer and the maximizer. In the diet problem, the minimizer has n 
foods (peanut butter and steak, in Section 8.1). They enter the diet in the (nonnegative) 
amounts x1,...,X,. The constraints represent m required vitamins, in place of the one 
earlier constraint of sufficient protein. The entry a;; measures the ith vitamin in the jth 
food, and the ith row of Ax > b forces the diet to include at least b; of that vitamin. If c; 
is the cost of the jth food, then cjx; +---+cy)x, = cx is the cost of the diet. That cost is 
to be minimized. 

In the dual, the druggist is selling vitamin pills at prices y; > 0. Since food j 
contains vitamins in the amounts a;;, the druggist’s price for the vitamin equivalent 
cannot exceed the grocer’s price c;. That is the jth constraint in yA < c. Working within 
this constraint on vitamin prices, the druggist can sell the required amount b; of each 
vitamin for a total income of yj bj +---+¥mbm = yb—to be maximized. 

The feasible sets for the primal and dual problems look completely different. The 
first is a subset of R”, marked out by x > 0 and Ax > b. The second is a subset of R”, 
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determined by y > 0 and A! and c. The whole theory of linear programming hinges on 
the relation between primal and dual. Here is the fundamental result: 


8D Duality Theorem When both problems have feasible vectors, they have 
optimal x* and y*. The minimum cost cx* equals the maximum income y*b. 


If optimal vectors do not exist, there are two possibilities: Either both feasible sets are 
empty, or one is empty and the other problem is unbounded (the maximum is + or the 
minimum is —co). 

The duality theorem settles the competition between the grocer and the druggist. The 
result is always a tie. We will find a similar “minimax theorem” in game theory. The 
customer has no economic reason to prefer vitamins over food, even though the druggist 
guarantees to match the grocer on every food—and even undercuts on expensive foods 
(like peanut butter). We will show that expensive foods are kept out of the optimal diet, 
so the outcome can be (and is) a tie. 

This may seem like a total stalemate, but I hope you will not be fooled. The optimal 
vectors contain the crucial information. In the primal problem, x* tells the purchaser 
what to buy. In the dual, y* fixes the natural prices (shadow prices) at which the economy 
should run. Insofar as our linear model reflects the true economy. x* and y* represent 
the essential decisions to be made. 

We want to prove that c*x = y*b. It may seem obvious that the druggist can raise 
the vitamin prices y* to meet the grocer, hut only one thing is truly clear: Since each 
food can be replaced by its vitamin equivalent, with no increase in cost, all adequate 
food diets must cost at least as much as vitamins. This is only a one-sided inequality, 
druggist’s price < grocer’s price. It is called weak duality, and it is easy to prove for 
any linear program and its dual: 


8E If x and yare feasible in the primal and dual problems, then yb < cx. 


Proof. Since the vectors are feasible, they satisfy Ax > b and yA < c. Because feasi- 
bility also includes x > 0 and y > 0, we can take inner products without spoiling those 
inequalities (multiplying by negative numbers would reverse them): 


yAx > yb and yAx < cx. (1) 


Since the left-hand sides are identical, we have weak duality yb < cx. 


This one-sided inequality prohibits the possibility that both problems are unbounded. 
If yb is arbitrarily large, a feasible x would contradict yb < cx. Similarly, if cx can go 
down to —ce, the dual cannot admit a feasible y. 

Equally important, any vectors that achieve yb = cx must be optimal. At that point 
the grocer’s price equals the druggist’s price. We recognize an optimal food diet and 
optimal vitamin prices by the fact that the consumer has nothing to choose: 


8F If the vectors x and y are feasible and cx = yb, then x and y are optimal. 
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Since no feasible y can make yb larger than cx, our y that achieves this value is opti- 
mal. Similarly, any x that achieves the cost cx = yb must be an optimal x*. 

We give an example with two foods and two vitamins. Note how A! appears when 
we write out the dual, since yA < c for row vectors means Aly! < c! for columns. 


Primal Minimize x; +4x2 Dual Maximize 6y,;+7y2 
subject to x; >0, x2 >0 subjectto y,; >0, yo>0 

2x; +x2 >6 2y,+5y2 <1 

5x1 +3x2 > 7. yy +3x2 <4. 


Solution x, = 3 and x2 = 0 are feasible, with cost x; + 4x2 = 3. In the dual, yj = 5 and 
y2 = 0 give the same value 6y; + 7y2 = 3. These vectors must be optimal. 


Please look closely to see what actually happens at the moment when yb = cx. Some 
of the inequality constraints are tight, meaning that equality holds. Other constraints are 
loose, and the key rule makes economic sense: 


(i) The diet has x = 0 when food j is priced above its vitamin equivalent. 
(ii) The price is y; = 0 when vitamin i is oversupplied in the diet x". 


In the example, x2 = 0 because the second food is too expensive. Its price exceeds the 
druggist’s price, since yj + 3y2 < 4 is a strict inequality 5+ 0 <4. Similarly, the diet 
required seven units of the second vitamin, but actually supplied 5x; + 3x2 = 15. So we 
found yz = 0, and that vitamin is a free good. You can see how the duality has become 
complete. 

These optimality conditions are easy to understand in matrix terms. From equation 
(1) we want y*Ax* = y*b at the optimum. Feasibility requires Ax* > b, and we look for 
any components in which equality fails. This corresponds to a vitamin that is oversup- 
plied, so its price is y; = 0. 

At the same time, we have y*A <c. All strict inequalities (expensive foods) corre- 
spond to x; = 0 (omission from the diet). That is the key to y"Ax* = cx", which we 
need. These are the complementary slackness conditions of linear programming, and 
the Kuhn-Tucker conditions of nonlinear programming: 


8G _ The optimal vectors x* and y* satisfy complementary slackness: 


If (Ax*);>b; then y; =0 If (y"A);>c; then xj =0. (2) 


Let me repeat the proof. Any feasible vectors x and y satisfy weak duality: 
yb < y(Ax) = (yA)x < cx. (3) 


We need equality, and there is only one way in which y*b can equal y*(Ax*). Any time 
bj < (Ax*);, the factor y* that multiplies these components must be zero. 
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Similarly, feasibility gives yAx < cx. We get equality only when the second slackness 
condition is fulfilled. If there is an overpricing (y*A) ; < c;, it must be canceled through 
multiplication by x; = 0. This leaves us with y"b = cx” in equation (3). This equality 
guarantees the optimality of x* and y*. 


The Proof of Duality 


The one-sided inequality yb < cx was easy to prove; it gave a quick test for optimal 
vectors (they turn it into an equality); and now it has given the slackness conditions in 
equation (2). The only thing it has not done is to show that y*b = cx* is really possible. 
Until those optimal vectors are actually produced, the duality theorem is not complete. 

To produce y* we return to the simplex method—which has already computed x*. Our 
problem is to show that the method stopped in the right place for the dual problem (even 
though it was constructed to solve the primal). Recall that the m inequalities Ax > b 
were changed to equations by introducing the slack variables w = Ax — b: 


Primal feasibility 4 7 E > 0. (4) 
w 


=b and E 
Ww 


Every simplex step picked m columns of the long matrix [A — I] to be basic, and shifted 
them (theoretically) to the front. This produced [B N]. The same shift reordered the 
long cost vector [c 0] into [cg cy]. The stopping condition, which brought the simplex 
method to an end, was r= cy —cpB-'N > 0. 

This condition r > 0 was finally met, since the number of corners is finite. At that 
moment the cost was as low as possible: 


=1 


Minimum cost C= lew cn| | 0 


b 
| =cpB 'b. (5) 
If we can choose y* = cgB™' in the dual, we certainly have y*b = cx*. The minimum 
and maximum will be equal. We have to show that this y* satisfies the dual constraints 
yA <cand y => 0: 
Dual feasibility [A -1] < E 0| . (6) 


When the simplex method reshuffles the long matrix and vector to put the basic variables 
first, this rearranges the constraints in equation (6) into 


y[B N|<|ce en]. (7) 


For y* = cpB_!, the first half is an equality and the second half is cpB-!N < cy. This is 
the stopping condition r > 0 that we know to be satisfied! Therefore our y* is feasible, 
and the duality theorem is proved. By locating the critical m by m matrix B, which is 
nonsingular as long as degeneracy is forbidden, the simplex method has produced the 
optimal y* as well as x*. 
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Shadow Prices 


In calculus, everybody knows the condition for a maximum or a minimum: The first 
derivatives are zero. But this is completely changed by constraints. The simplest exam- 
ple is the line y = x. Its derivative is never zero, calculus looks useless, and the largest 
y is certain to occur at the end of the interval. That is exactly the situation in linear pro- 
gramming! There are more variables, and an interval is replaced by a feasible set, but 
still the maximum is always found at a corner of the feasible set (with only m nonzero 
components). 


The problem in linear programming is to locate that cornet For this, calculus is not 
completely helpless. Far from it, because “Lagrange multipliers” will bring back zero 
derivatives at the maximum and minimum. The dual variables y are exactly the La- 
grange multipliers. And they answer the key question: How does the minimum cost 
cx* = y*b change, if we change D or c? 

This is a question in sensitivity analysis. It allows us to squeeze extra information out 
of the dual problem. For an economist or an executive, these questions about marginal 
cost are the most important. 


If we allow large changes in b or c, the solution behaves in a very jumpy way. As 
the price of eggs increases, there will be a point at which they disappear from the diet. 
The variable xegg will jump from basic to free. To follow it properly, we would have 
to introduce “parametric” programming. But if the changes are small, the corner that 
was optimal remains optimal. The choice of basic variables does not change; B and N 
stay the same. Geometrically, we shifted the feasible set a little (by changing b), and we 
tilted the planes that come up to meet it (by changing c). When these changes are small, 
contact occurs at the same (slightly moved) corner. 


At the end of the simplex method, when the right basic variables are known, the 
corresponding m columns of A make up the basis matrix B. At that corner, a shift of size 
Ab changes the minimum cost by y*Ab. The dual solution y* gives the rate of change 
of minimum cost (its derivative) with respect to changes in b. The components of y* 
are the shadow prices. If the requirement for a vitamin goes up by A, and the druggist’s 
price is y;, then the diet cost (from druggist or grocer) will go up by y|A. In the case 
that y| is zero, that vitamin is a free good and the small change has no effect. The diet 
already contained more than b. 


We now ask a different question. Suppose we insist that the diet contain some small 
edible amount of egg. The condition xegg > 0 is changed to xegg > 6. How does this 
change the cost? 


* 


If eggs were in the diet x", there is no change. But if x-,, = 0, it will cost extra to add 
in the amount 6. The increase will not be the full price Cege9, since we can cut down on 
other foods. The reduced cost of eggs is their own price, minus the price we are paying 
for the equivalent in cheaper foods. To compute it we return to equation (2) of Section 
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8.2: 
cost = (cy — cpB 'N)xy +cpB-!b=rxy +cpB'b. 


If egg is the first free variable, then increasing the first component of xy to 6 will increase 
the cost by r,6. The real cost of egg is r,. This is the change in diet cost as the zero lower 
bound (nonnegativity constraint) moves upwards. We know that r > 0, and economics 
tells us the same thing: The reduced cost of eggs cannot be negative or they would have 
entered the diet. 


Interior Point Methods 


The simplex method moves along edges of the feasible set, eventually reaching the opti- 
mal corner x*. Interior point methods start inside the feasible set (where the constraints 
are all inequalities). These methods hope to move more directly to x* (and also find y*). 
When they are very close to the answer, they stop. 

One way to stay inside is to put a barrier at the boundary. Add an extra cost in the 
form of a logarithm that blows up when any variable x or any slack variable w = Ax —b 
touches zero. The number 0 is a small parameter to be chosen: 


n m 
Barrier problem P(6@) Minimize cx—@ (x Inx; + Yin “ . (8) 
1 1 


This cost is nonlinear (but linear programming is already nonlinear, from inequalities). 
The notation is simpler if the long vector (x,w) is renamed x and [A — J] is renamed A. 
The primal constraints are now x > 0 and Ax = b. The sum of Inx; in the barrier now 
goes tom-+n. 

The dual constraints are yA < c. (We don’t need y > 0 when we have Ax = b in 
the primal.) The slack variable is s = c — yA, with s > 0. What are the Kuhn-Tucker 
conditions for x and y to be the optimal x* and y*? Along with the constraints we require 
duality: cx* = y*b. 

Including the barrier gives an approximate problem P(@). For its Kuhn-Tucker op- 
timality conditions, the derivative of Inx; gives 1/x;. If we create a diagonal matrix X 
from those positive numbers x;, and use e= [1 --- 1] for the row vector of n+ m ones, 
then optimality in P(@) is as follows: 


Primal (column vectors) Ax=b with x>0 (9a) 
Dual (row vectors) yA + 0ex “lig (9b) 


As 8 — 0, we expect those optimal x and y to approach x* and y* for the original no- 
barrier problem, and @eX~! will stay nonnegative. The plan is to solve equations (9a— 
9b) with smaller and smaller barriers, given by the size of 0. 

In reality, those nonlinear equations are approximately solved by Newton’s method 
(which means they are linearized). The nonlinear term is s = @eX~!. To avoid 1/x;, 
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rewrite that as sX = Oe. Creating the diagonal matrix S from s, this is eSX = Oe. If we 
change e, y, c, and s to column vectors, and transpose, optimality now has three parts: 


Primal Ax=b, x>0. (10a) 
Dual Alyts=c. (10b) 
Nonlinear XSe—@0e=0. (10c) 


Newton’s method takes a step Ax, Ay, As from the current x, y, s. (Those solve equa- 
tions (10a) and (10b), but not (10c).) By ignoring the second-order term AX ASe, the 
corrections come from linear equations! 


AAx = 0. (11a) 
Newton step AlAy+As = 0. (11b) 
SAx+ XAs = Oe —XSe. (11c) 


Robert Freund’s notes for his MIT class pin down the (quadratic) convergence rate and 
the computational complexity of this algorithm. Regardless of the dimensions m and n, 
the duality gap sx is generally below 10~® after 20-80 Newton steps. This algorithm 
is used almost “as is” in commercial interior-point software, and for a large class of 
nonlinear optimization problems as well. 


The Theory of Inequalities 


There is more than one way to study duality. We quickly proved yb < cx, and then 
used the simplex method to get equality. This was a constructive proof; x* and y* were 
actually computed. Now we look briefly at a different approach, which omits the simplex 
algorithm and looks more directly at the geometry. I think the key ideas will be just as 
clear (in fact, probably clearer) if we omit some of the details. 

The best illustration of this approach came in the Fundamental Theorem of Linear 
Algebra. The problem in Chapter 2 was to find b in the column space of A. After elim- 
ination and the four subspaces, this solvability question was answered in a completely 
different way by Problem 11 in Section 3.1: 


8H Ax=DbDhasasolution or there is ay such that yA =0 and yb 40. 


This is the theorem of the alternative, because to find both x and y is impossible: If Ax = 
b then yAx = yb ¥ 0, and this contradicts yAx = Ox = 0. In the language of subspaces, 
either b is in the column space, or it has a component sticking into the left nullspace. 
That component is the required y. 

For inequalities, we want to find a theorem of exactly the same kind. Start with the 
same system Ax = b, but add the constraint x > 0. When does there exist a nonnegative 
solution to Ax = b? 

In Chapter 2, b was anywhere in the column space. Now we allow only nonnegative 
combinations, and the b’s no longer fill out a subspace. Instead, they fill a cone-shaped 
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P 
column 1 separating 
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column 2 , 
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bis in b is . 
the cone outside 
the cone 


Figure 8.4: The cone of nonnegative combinations of the columns: b = Ax with x > 0. When d is outside the cone, 
it is separated by a hyperplane (perpendicular to y). 


region. For n columns in R”, the cone becomes an open-ended pyramid. Figure 8.4 has 
four vectors in R’, and A is 2 by 4. If b lies in this cone, there is a nonnegative solution 
to Ax = b; otherwise not. 

What is the alternative if b lies outside the cone? Figure 8.4 also shows a “separating 
hyperplane,’ which has the vector b on one side and the whole cone on the other side. 
The plane consists of all vectors perpendicular to a fixed vector y. The angle between 
y and b is greater than 90°, so yb < 0. The angle between y and every column of A is 
less than 90°, so yA > 0. This is the alternative we are looking for. This theorem of the 
separating hyperplane is fundamental to mathematical economics. 


8l Ax =b has a nonnegative solution or there is a y with yA > 0 and 
yb <0. 


Example 1. The nonnegative combinations of the columns of A = / fill the positive 
quadrant b > 0. For every other b, the alternative must hold for some y: 


2 
Not in cone o=| | then y= [0 1 gives yI>0 but yb=-3. 


The x-axis, perpendicular to y = [0 1], separates b from the cone = quadrant. 


Here is a curious pair of alternatives. It is impossible for a subspace S and its or- 
thogonal complement $+ both to contain positive vectors. Their inner product would 
be positive, not zero. But S might be the x-axis and S+ the y-axis, in which case they 
contain the “semipositive” vectors [1 0] and [0 1]. This slightly weaker alternative does 
work: Either S contains a positive vector x > 0, or S* contains a nonzero y > 0. When S 
and S+ are perpendicular lines in the plane, one or the other must enter the first quadrant. 
I can’t see this clearly in three or four dimensions. 
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For linear programming, the important alternatives come when the constraints are 


inequalities. When is the feasible set empty (no x)? 


8J Ax>bhasasolutionx>0 or thereisay<0Owith yA >0 and yb <0. 


Proof. The slack variables w = Ax — b change Ax > b into an equation. Use 8I: 


First alternative 4 — ; 
w 


=b_ for some | > 0. 
w 


Second alternative y [A -1] 3 lo 0| forsome y with yb< 0. 


It is this result that leads to a “nonconstructive proof” of the duality theorem. 


Problem Set 8.3 


ki 


What is the dual of the following problem: Minimize x; + x2, subject to x; > 0, 
x2 > 0, 2x, > 4, x1 + 3x2 > 11? Find the solution to both this problem and its dual, 
and verify that minimum equals maximum. 


. What is the dual of the following problem: Maximize y2 subject to y; > 0, y2 > 0, 


y1 +y2 <3? Solve both this problem and its dual. 


Suppose A is the identity matrix (so that m = n), and the vectors b and c are nonnega- 
tive. Explain why x* = b is optimal in the minimum problem, find y* in the maximum 
problem, and verify that the two values are the same. If the first component of b is 
negative, what are x* and y*? 


Construct a 1 by 1 example in which Ax > b, x > 0 is unfeasible, and the dual problem 
is unbounded. 


Starting with the 2 by 2 matrix A = [4 | , choose b and c so that both of the feasible 
sets Ax > b, x > 0 and yA <c, y > 0 are empty. 


If all entries of A, b, and c are positive, show that both the primal and the dual are 
feasible. 


Show that x = (1,1,1,0) and y = (1,1,0,1) are feasible in the primal and dual, with 


0 
0 
1 
1 


nO —_- — — 


Then, after computing cx and yb, explain how you know they are optimal. 


10. 


11. 


12. 


13. 


14. 


15. 


16. 


17. 


18. 
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Verify that the vectors in the previous exercise satisfy the complementary slackness 
conditions in equation (2), and find the one slack inequality in both the primal and 
the dual. 


Suppose that A= [} 9], b= [',], and c = [1]. Find the optimal x and y, and verify 
the complementary slackness conditions (as well as yb = cx). 


If the primal problem is constrained by equations instead of inequalities—Minimize 
cx subject to Ax = b and x > 0—then the requirement y > 0 is left out of the dual: 
Maximize yb subject to yA < c. Show that the one-sided inequality yb < cx still 
holds. Why was y > 0 needed in equation (1) but not here? This weak duality can be 
completed to full duality. 


(a) Without the simplex method, minimize the cost 5x; + 3x2 + 4x3, subject to x; + 
x2 +%3 > 1, x1 > 0, x2 > 0, x3 > 0. 


(b) What is the shape of the feasible set? 


(c) What is the dual problem, and what is its solution y? 


If the primal has a unique optimal solution x*, and then c is changed a little, explain 
why x* still remains the optimal solution. 


Write the dual of the following problem: Maximize x; +x2+ x3 subject to 2x; +x2 < 
4, x3 <6. What are the optimal x* and y* (if they exist!)? 


IfA= EK 1] , describe the cone of nonnegative combinations of the columns. If b lies 
inside that cone, say b = (3,2), what is the feasible vector x? If b lies outside, say 
b = (0,1), what vector y will satisfy the alternative? 


In three dimensions, can you find a set of six vectors whose cone of nonnegative 
combinations fills the whole space? What about four vectors? 


Use 8H to show that the following equation has no solution, because the alternative 


holds: 
2 2 1 
<= 
4 4 1 
Use 8I to show that there is no solution x > 0 (the alternative holds): 
1 3 -5 2 
= 
1 -4 -7 3 


Show that the alternatives in 8J (Ax > b, x > 0, yA > 0, yb < 0, y < 0) cannot both 
hold. Hint: yAx. 
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8.4 Network Models 


Some linear problems have a structure that makes their solution very quick. Band ma- 
trices have all nonzeros close to the main diagonal, and Ax = b is easy to solve. In linear 
programming, we are interested in the special class for which A is an incidence matrix. 
Its entries are —1 or +1 or (mostly) zero, and pivot steps involve only additions and 
subtractions. Much larger problems than usual can be solved. 

Networks enter all kinds of applications. Traffic through an intersection satisfies 
Kirchhoff’s current law: flow in equals flow out. For gas and oil, network programming 
has designed pipeline systems that are millions of dollars cheaper than the intuitive (not 
optimized) designs. Scheduling pilots and crews and airplanes has become a significant 
problem in applied mathematics! We even solve the marriage problem—to maximize 
the number of marriages when brides have a veto. That may not be the real problem, but 
it is the one that network programming solves. 

The problem in Figure 8.5 is to maximize the flow from the source to the sink. The 
flows cannot exceed the capacities marked on the edges, and the directions given by 
the arrows cannot be reversed. The flow on the two edges into the sink cannot exceed 
6+ 1=7. Is this total of 7 achievable? What is the maximal flow from left to right? 

The unknowns are the flows x;; from node i to node j. The capacity constraints are 
Xjj <cjj. The flows are nonnegative: x;; > 0 going with the arrows. By maximizing the 
return flow x6; (dotted line), we maximize the total flow into the sink. 


cut. 

capacity 

2+34+1,\ 
\ 


SOUTCE 


4 


Figure 8.5: A 6-node network with edge capacities: the maximal flow problem. 


Another constraint is still to be heard from. It is the “conservation law,” that the flow 
into each node equals the flow out. That is Kirchhoff’s current law: 


Current law Ya ra tore Gade eng. (12) 
i k 
The flows x;; enter node j from earlier nodes i. The flows xj, leave node j to later 


nodes k. The balance in equation (1) can be written as Ax = 0, where A is a node-edge 
incidence matrix (the transpose of Section 2.5). A has a row for every node and a +1, 


8.4 Network Models 445 


—1 column for every edge: 


1 1 —1 node | 
—1 | 1 2 
Incidence St —] 1 1 3 
Matrix —] —1 1 4 
—| —| 1 5 
—1 -1 1 6 

edge 12 13 24 25 34 35 46 56 61 


Maximal Flow Maximize x6; subject to Ax = 0 and 0 < xj; < cj;. 


A flow of 2 can go on the path 1-2-4-6-1. A flow of 3 can go along 1-3-4-6-1. An 
additional flow of 1 can take the lowest path 1-3-5-6-1. The total is 6, and no more is 
possible. How do you prove that the maximal flow is 6 and not 7? 

Trial and error is convincing, but mathematics is conclusive: The key is to find a cut in 
the network, across which all capacities are filled. That cut separates nodes 5 and 6 from 
the others. The edges that go forward across the cut have total capacity 2+ 3-+ 1 =6— 
and no more can get across! Weak duality says that every cut gives a bound to the total 
flow, and full duality says that the cut of smallest capacity (the minimal cut) is filled by 
the maximal flow. 


8K Max flow-min cut theorem. The maximal flow in a network equals the 
total capacity across the minimal cut. 


A “cut” splits the nodes into two groups S and T (source in S and sink in 7). Its capacity 
is the sum of the capacities of all edges crossing the cut (from S to T). Several cuts 
might have the same capacity. Certainly the total flow can never be greater than the total 
capacity across the minimal cut. The problem, here and in all of duality, is to show that 
equality is achieved by the right flow and the right cut. 


Proof that max flow = min cut. Suppose a flow is maximal. Some nodes might still be 
reached from the source by additional flow, without exceeding any capacities. Those 
nodes go with the source into the set S. The sink must lie in the remaining set 7, or it 
could have received more flow! Every edge across the cut must he filled, or extra flow 
could have gone further forward to a node in 7. Thus the maximal flow does fill this cut 
to capacity. and equality has been achieved. 


This suggests a way to construct the maximal flow: Check whether any path has 
unused capacity. If so, add flow along that “augmenting path.’ Then compute the re- 
maining capacities and decide whether the sink is cut off from the source, or additional 
flow is possible. If you label each node in S by the previous node that flow could come 
from, you can backtrack to find the path for extra flow. 
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The Marriage Problem 


Suppose we have four women and four men. Some of those sixteen couples are compat- 
ible, others regrettably are not. When is it possible to find a complete matching, with 
everyone married? If linear algebra can work in 20-dimensional space, it can certainly 
handle the trivial problem of marriage. 

There are two ways to present the problem—in a matrix or on a graph. The matrix 
contains a;; = 0 if the ith woman and jth man are not compatible, and a;; = 1 if they are 
willing to try. Thus row 7 gives the choices of the ith woman, and column j corresponds 
to the jth man: 


C tibilit 
ompatibility A= has 6 compatible pairs. 


matrix 


oO FR 
oor Oo 
Co OoOrF oO 
ee RP Oo OO 


The left graph in Figure 8.6 shows two possible marriages. Ignoring the source s and 
sink ¢, it has four women on the left and four men on the right. The edges correspond 
to the 1s in the matrix, and the capacities are | marriage. There is no edge between the 
first woman and fourth man, because the matrix has aj4 = 0. 


1 


Figure 8.6: Two marriages on the left, three (maximum) on the right. The third is created by adding two new 
marriages and one divorce (backward flow). 


It might seem that node M> can’t be reached by more flow—but that is not so! The 
extra flow on the right goes backward to cancel an existing marriage. This extra flow 
makes 3 marriages, which is maximal. The minimal cut is crossed by 3 edges. 

A complete matching (if it is possible) is a set of four is in the matrix. They would 
come from four different rows and four different columns, since bigamy is not allowed. 
It is like finding a permutation matrix within the nonzero entries of A. On the graph, this 
means four edges with no nodes in common. The maximal flow is less than 4 exactly 
when a complete matching is impossible. 

In our example the maximal flow is 3, not 4. The marriages 1—1, 2—2, 44 are allowed 
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(and several other sets of three marriages), but there is no way to reach four. The minimal 
cut on the right separates the two women at the bottom from the three men at the top. 
The two women have only one man left to choose—not enough. The capacity across the 
cut is only 3. 


Whenever there is a subset of k women who among them like fewer than k men, a 
complete matching is impossible. 


That test is decisive. The same impossibility can be expressed in different ways: 


1. (For Chess) It is impossible to put four rooks on squares with 1s in A, so that no 
rook can take any other rook. 


2. (For Marriage Matrices) The Is in the matrix can be covered by three horizontal 
or vertical lines. That equals the maximum number of marriages. 


3. (For Linear Algebra) Every matrix with the same zeros as A is singular. 


Remember that the determinant is a sum of 4! = 24 terms. Each term uses all four rows 
and columns. The zeros in A make all 24 terms zero. 

A block of zeros is preventing a complete matching! The 2 by 3 submatrix in rows 3, 
4 and columns 1, 2, 3 of A is entirely zero. The general rule for an n by n matrix is that 
a p by q block of zeros prevents a matching if p+ q > n. Here women 3, 4 could marry 
only the man 4. If p women can marry only n— gq men and p > n— gq (which is the same 
as a zero block with p+ q > n), then a complete matching is impossible. 

The mathematical problem is to prove the following: If every set of p women does 
like at least p men, a complete matching is possible. That is Hall’s condition. No block 
of zeros is too large. Each woman must like at least one man, each two women must 
between them like at least two men, and so on, to p = n. 


8L A complete matching is possible if (and only if) Hall’s condition holds. 


The proof is simplest if the capacities are n, instead of 1, on all edges across the 
middle. The capacities out of the source and into the sink are still 1. If the maximal flow 
is n, all those edges from the source and into the sink are filled—and the flow produces 
n marriages. When a complete matching is impossible, and the maximal flow is below 
n, some cut must be responsible. 

That cut will have capacity below n, so no middle edges cross it. Suppose p nodes on 
the left and r nodes on the right are in the set S with the source. The capacity across that 
cut is n — p from the source to the remaining women, and r from these men to the sink. 
Since the cut capacity is below n, the p women like only the r men and no others. But 
the capacity n — p +r is below n exactly when p > r, and Hall’s condition fails. 


448 Chapter 8 Linear Programming and Game Theory 
Spanning Trees and the Greedy Algorithm 


A fundamental network model is the shortest path problem—in which the edges have 
lengths instead of capacities. We want the shortest path from source to sink. If the edges 
are telephone lines and the lengths are delay times, we are finding the quickest route 
for a call, If the nodes are computers, we are looking for the perfect message-passing 
protocol. 

A closely related problem finds the shortest spanning tree—a set of n—1 edges 
connecting all the nodes of the network. Instead of getting quickly between a source 
and a sink, we are now minimizing the cost of connecting all the nodes. There are no 
loops, because the cost to close a loop is unnecessary. A spanning tree connects the 
nodes without loops, and we want the shortest one. Here is one possible algorithm: 


1. Start from any node s and repeat the following step: 


Add the shortest edge that connects the current tree to a new node. 


In Figure 8.7, the edge lengths would come in the order 1, 2, 7, 4, 3, 6. The last step 
skips the edge of length 5, which closes a loop. The total length is 23—but is it minimal? 
We accepted the edge of length 7 very early, and the second algorithm holds out longer. 


3 next? 


Figure 8.7: A network and a shortest spanning tree of length 23. 


2. Accept edges in increasing order of length, rejecting edges that complete a loop. 


Now the edges come in the order 1, 2, 3, 4, 6 (again rejecting 5), and 7. They are the 
same edges—although that will not always happen. Their total length is the same—and 
that does always happen. The spanning tree problem is exceptional, because it can be 
solved in one pass. 

In the language of linear programming, we are finding the optimal corner first. The 
spanning tree problem is being solved like back-substitution, with no false steps. This 
general approach is called the greedy algorithm. Here is another greedy idea: 


3. Build trees from all n nodes, by repeating the following step: 


Select any tree and add the minimum-length edge going out from that tree. 


The steps depend on the selection order of the trees. To stay with the same tree is 
algorithm 1. To take the lengths in order is algorithm 2. To sweep through all the trees 
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in turn is a new algorithm. It sounds so easy, but for a large problem the data structure 
becomes critical, With a thousand nodes, there might be nearly a million edges, and you 
don’t want to go through that list a thousand times. 


Further Network Models 


There are important problems related to matching that are almost as easy: 


1. The optimal assignment problem: a;; measures the value of applicant i in job j. 
Assign jobs to maximize the total value—the sum of the a;; on assigned jobs. (If 
all a;; are O or 1, this is the marriage problem.) 


2. The transportation problem: Given supplies at n points and demands at n markets 
choose shipments x;; from suppliers to markets that minimize the total cost )' C;;x;;. 
(If all supplies and demands are 1, this is the optimal assignment problem—sending 
one person to each job.) 


3. Minimum costflow: Now the routes have capacities c;; as well as costs C;;, mixing 
the maximal flow problem with the transportation problem. What is the cheapest 
flow, subject to capacity constraints? 


A fascinating part of this subject is the development of algorithms. Instead of a 
theoretical proof of duality, we use breadth-first search or depth-first search to find 
the optimal assignment or the cheapest flow. It is like the simplex method, in starting 
from a feasible flow (a corner) and adding a new flow (to move to the next corner), The 
algorithms are special because network problems involve incidence matrices. 

The technique of dynamic programming rests on a simple idea: If a path from source 
to sink is optimal, then each part of the path must be optimal. The solution is built 
backwards from the sink, with a multistage decision process. At each stage, the distance 
to the sink is the minimum of a new distance plus an old distance: 


Bellman equation x-t distance = minimum over y of (x-y + y-t distances). 


I wish there were space for more about networks. They are simple but beautiful. 


Problem Set 8.4 


1. In Figure 8.5, add 3 to every capacity. Find by inspection the maximal flow and 
minimal cut. 


2. Find a maximal flow and minimal cut for the following network: 
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10. 
11. 
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. If you could increase the capacity of any one pipe in the network above, which 


change would produce the largest increase in the maximal flow? 


Draw a 5-node network with capacity |i — j| between node i and node j. Find the 
largest possible flow from node 1 to node 4. 


In a graph, the maximum number of paths from s to tf with no common edges equals 
the minimum number of edges whose removal disconnects s from t. Relate this to 
the max flow-min cut theorem. 


Find a maximal set of marriages (a complete matching, if possible) for 


0010 0 be 30 Oe O 
ESE Qe dt. 010 1 0 
A=|01101 and B=|0 0101 
001 1 0 od 10) 0 
00010 100 0 0 


Sketch the network for B, with heavier lines on the edges in your matching. 


For the matrix A in Problem 6, which rows violate Hall’s condition—by having all 
their 1s in too few columns? Which p by qg submatrix of zeros has p+q > n? 


. How many lines (horizontal and vertical) are needed to cover all the 1s in A in Prob- 


lem 6? For any matrix, explain why weak duality is true: If k marriages are possible, 
then it takes at least k lines to cover all the Is. 


(a) Suppose every row and every column contains exactly two Is. Prove that a com- 
plete matching is possible. (Show that the 1s cannot be covered by less than n 
lines) 


(b) Find an example with two or more is in each row and column, for which a com- 
plete matching is impossible. 


If a7 by 7 matrix has 15 1s, prove that it allows at least 3 marriages. 


For infinite sets, a complete matching may be impossible even if Hail’s condition is 
passed. If the first row is all 1s and then every a; ;—-; = 1, show that any p rows have 
1s in at least p columns—and yet there is no complete matching. 
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12. If Figure 8.5 shows lengths instead of capacities, find the shortest path from s to f, 
and a minimal spanning tree. 


13. Apply algorithms 1 and 2 to find a shortest spanning tree for the network of Problem 
Zi 
14. (a) Why does the greedy algorithm work for the spanning tree problem? 
(b) Show by example that the greedy algorithm could fail to find the shortest path 
from s to t, by starting with the shortest edge. 
15. If A is the 5 by 5 matrix with is just above and just below the main diagonal, find 


(a) a set of rows with Is in too few columns. 
(b) a set of columns with is in too few rows. 
(c) a p by q submatrix of zeros with p+q> 5. 
(d) four lines that cover all the Is. 


16. The maximal flow problem has slack variables w;; = ci; —x;; for the difference be- 
tween capacities and flows. State the problem of Figure 8.5 as a linear program. 


8.5 Game Theory 


The best way to explain a two-person zero-sum game is to give an example. It has two 
players X and Y, and the rules are the same for every turn: 


X holds up one hand or two, and so does Y. If they make the same decision, Y 
wins $10. If they make opposite decisions, X wins $10 for one hand and $20 


for two: 
Payoff matrix pe —10 20 one hand by Y 
(payments to X) = 10 —10 two hands by Y 


one hand two hands 
by X by X 


If X does the same thing every time, Y will copy him and win. Similarly Y cannot stick 
to a single strategy, or X will do the opposite. Both players must use a mixed strategy, 
and the choice at every turn must be independent of the previous turns. If there is some 
historical pattern, the opponent can take advantage of it. Even the strategy “stay with 
the same choice until you lose” is obviously fatal. After enough plays, your opponent 
would know exactly what to expect. 

In a mixed strategy, X can put up one hand with frequency x; and both hands with 
frequency x7 = 1—.x,. At every turn this decision is random. Similarly Y can pick 
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probabilities y; and y2 = 1 — y;. None of these probabilities should be 0 or 1; otherwise 
the opponent adjusts and wins. If they equal 4, Y would be losing $20 too often. (He 
would lose $20 a quarter of the time, $10 another quarter of the time, and win $10 half 
the time—an average loss of $2.50. This is more than necessary.) But the more Y moves 
toward a pure two-hand strategy, the more X will move toward one hand. 

The fundamental problem is fo find the best mixed strategies. Can X choose probabil- 
ities x; and x2 that present Y with no reason to move his own strategy (and vice versa)? 
Then the average payoff will have reached a saddle point: It is a maximum as far as X 
is concerned, and a minimum as far as Y is concerned. To find such a saddle point is to 
solve the game. 

X is combining the two columns with weights x; and | — x; to produce a new “mixed” 
column. Weights 2 and 2 would produce this column: 


3 |-10 21 20 2 
Mi ] = = = L 
ixed column eo) +3 (2 A 


Against this mixed strategy, Y will always lose $2. This does not mean that all 
strategies are optimal for Y! If Y is lazy and stays with one hand, X will change and 
start winning $20. Then Y will change, and then X again. Finally, since we assume they 
are both intelligent, they settle down to optimal mixtures. Y will combine the rows with 
weights y; and 1 — y;, trying to produce a new row which is as small as possible: 


Mixed row v1 |=10 20] + (191) [10 —10| = |10-20y, ~10+30y, |. 


The right mixture makes the two components equal, at yj = z. Then both components 
equal 2; the mixed row becomes [2 2]. With this strategy Y cannot lose more than $2. 
Y has minimized the maximum loss, and that minimax agrees with the maximin found 
by X. The value of the game is minimax = maximin = $2. 

The optimal mixture of rows might not always have equal entries! Suppose X is 
allowed a third strategy of holding up three hands to win $60 when Y puts up one hand 
and $80 when Y puts up two. The payoff matrix becomes 


A= 
10 —10 80 


-~10 20 A 


X will choose the three-hand strategy (column 3) every time, and win at least $60. At 
the same time, Y always chooses the first row; the maximum loss is $60. We still have 
maximin = minimax = $60, but the saddle point is over in the corner. 

In Y’s optimal mixture of rows, which was purely row 1, $60 appears only in the 
column actually used by X. In X’s optimal mixture of columns, which was column 3, 
$60 appears in the row that enters Y’s best strategy. This rule corresponds exactly to the 
complementary slackness condition of linear programming. 


8.5 Game Theory 453 


Matrix Games 


The most general “m by n matrix game” is exactly like our example. X has n possible 
moves (columns of A). Y chooses from the m rows. The entry a;; is the payment when 
X chooses column j and Y chooses row i. A negative entry means a payment to Y. This 
is a zero-sum game. Whatever one player loses, the other wins. 


X is free to choose any mixed strategy x = (x1,...,X,). These x; give the frequencies 
for the n columns and they add to 1. At every turn X uses a random device to produce 
strategy i with frequency x;. Y chooses a vector y = (y1,.--,¥m), also with y; > 0 and 


yy; = 1, which gives the frequencies for selecting rows. 

A single play of the game is random. On the average, the combination of column j 
for X and row i for Y will turn up with probability x;y;. When it does come up, the payoff 
is a;j. The expected payoff to X from this combination is a;;x;y;, and the total expected 
payoff from each play of the same game is Y\Y\ a; jx jy; = yAx: 


xX] 
aij aj2 +7" Ain 
) X2) = AyiX1V1 + +++ + AmnXnYm 
yAx = |y1 Ym 
= average payoff. 
Qm1 Gm2 °*** Amn 
Xn 


It is this payoff yAx that X wants to maximize and Y wants to minimize. 


Example 1. Suppose A is the 1 by n identity matrix, A = 7. The expected payoff be- 
comes ylx = x1yj +-+-+Xnyn. X is hoping to hit on the same choice as Y, to win aj; = $1. 
Y is trying to evade X, to pay a;; = $0. If X chooses any column more often than another, 
Y can escape more often. The optimal mixture is x* = (1/n,1/n,...,1/n). Similarly Y 
cannot overemphasize any row—the optimal mixture is y* = (1/n,1/n,...,1/n). The 
probability that both will choose strategy i is (1/n)”, and the sum over i is the expected 
payoff to X. The total value of the game is n times (1/n)?, or 1/n: 


1 1/n 


yAx* = |L/n ss ibe Se -()+-+(4)-4 


1] |1/n 


As n increases, Y has a better chance to escape. The value 1 /n goes down. 

The symmetric matrix A = / did not make the game fair. A skew-symmetric matrix, 
A! = —A, means a completely fair game. Then a choice of strategy j by X and i by Y 
wins aj; for X, and a choice of j by Y andi by X wins the same amount for Y (because 
aji = —aj;;j). The optimal strategies x* and y* must be the same, and the expected payoff 
must be y*Ax* = 0. The value of the game, when A! = —A, is zero. But the strategy is 
still to be found. 
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Example 2. 
0 -1 -l 
Fair game A=!/1 0 -l 
1 1 O 


In words, X and Y both choose a number between | and 3. The smaller choice wins $1. 
(If X chooses 2 and Y chooses 3, the payoff is a32 = $1; if they choose the same number, 
we are on the diagonal and nobody wins.) Neither player can choose a strategy involving 
2 or 3. The pure strategies x* = y* = (1,0,0) are optimal—both players choose 1 every 
time. The value is y*Ax* = a1, = 0. 


The matrix that leaves all decisions unchanged has mn equal entries, say @. This 
simply means that X wins an additional amount @ at every turn. The value of the game 
is increased by @, but there is no reason to change x* and y*. 


The Minimax Theorem 


Put yourself in the place of X, who chooses the mixed strategy x = (x1,...,Xn). Y will 
eventually recognize that strategy and choose y to minimize the payment yAx. An intel- 
ligent player X will select x* to maximize this minimum: 


X wins at least min yAx* = max minyAx. (1) 
y xy 


Player Y does the opposite. For any chosen strategy y, X will maximize yAx. There- 
fore Y will choose the mixture y* that minimizes this maximum: 
Y loses no more than max y’Ax = minmax yAx. (2) 
x y x 
I hope you see what the key result will be, if it is true. We want the amount in 
equation (1) that X is guaranteed to win to equal the amount in equation (2) that Y must 
be satisfied to lose. Then the game will be solved: X can only lose by moving from x* 
and Y can only lose by moving from y*, The existence of this saddle point was proved 
by von Neumann: 


8M_ For any matrix A, the minimax over all strategies equals the maximin: 


Minimax theorem max min yAx = minmax yAx = value of the game. 
Xx 


y woe 
(3) 

If the maximum on the left is attained at x*, and the minimum on the right is 

attained at y*, this is a saddle point from which nobody wants to move: 


yAx < y"Ax* < yAx* for all x and y. (4) 


At this saddle point, x* is at least as good as any other x (since y*Ax < y*Ax*). And the 
second player Y could only pay more by leaving y*. 
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As in duality theory, maximin < minimax is easy. We combine the definition in equa- 
tion (1) of x* and the definition in equation (2) of y*: 


max min yAx = min yAx* < y*Ax* < max y*Ax = minmax yAx. (5) 
x y y x y x 


This only says that if X can guarantee to win at least @, and Y can guarantee to lose no 
more than B, then a < B. The achievement of von Neumann was to prove that a = B. 
The minimax theorem means that equality must hold throughout equation (5). 

For us, the striking thing about the proof is that it uses exactly the same mathematics 
as the theory of linear programming. X and Y are playing “dual” roles. They are both 
choosing strategies from the “feasible set” of probability vectors: x; > 0, )}x; = 1, y; = 0, 
yy; = 1. What is amazing is that even von Neumann did not immediately recognize the 
two theories as the same. (He proved the minimax theorem in 1928, linear programming 
began before 1947, and Gale, Kuhn, and Tucker published the first proof of duality 
in 1951—based on von Neumann’s notes!) We are reversing history by deducing the 
minimax theorem from duality. 

Briefly, the minimax theorem can be proved as follows. Let b be the column vector 
of m 1s, and c be the row vector of n 1s. These linear programs are dual: 


(P) minimize cx (D) maximize yb 
subject to Ax > b,x > 0 subject to yA <c, y>0. 


To make sure that both problems are feasible, add a large number @ to all entries of 
A. This cannot affect the optimal strategies, since every payoff goes up by a. For the 
resulting matrix, which we still denote by A, y = 0 is feasible in the dual and any large x 
is feasible in the primal. 

The duality theorem of linear programming guarantees optimal x* and y* with cx* = 
y’b. Because of the 1s in b and c, this means that Y x; = Py; = S. Division by S changes 
the sums to 1—and the resulting mixed strategies x*/S and y*/S are optimal. For any 
other strategies x and y, 


Ax >b implies yAx*>yb=1 and y'A<c implies y'Ax<cx=1. 


The main point is that y*Ax < 1 < yAx*. Dividing by S, this says that player X cannot 
win more than 1/S against the strategy y*/S, and player Y cannot lose less than 1/S 
against x*/S. Those strategies give maximin = minimax = 1/S. 


Real Games 


This completes the theory, but it leaves a natural question: Which ordinary games are 
actually equivalent to “matrix games”? Do chess and bridge and poker fit into von 
Neumann’s theory? 

I think chess does not fit very well, for two reasons. A strategy for black must include 
a decision on how to respond to white’s first play, and second play, and so on to the end 
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of the game. X and Y have billions of pure strategies. I do not see much of a role for 
chance. If white can find a winning strategy or if black can find a drawing strategy— 
neither has ever been found—that would effectively end the game of chess. You could 
play it like tic-tac-toe, but the excitement would go away. 

Bridge does contain some deception—as in a finesse. It counts as a matrix game, but 
m and n are again fantastically big. Perhaps separate parts of bridge could be analyzed 
for an optimal strategy. The same is true in baseball, where the pitcher and batter try 
to outguess each other on the choice of pitch. (Or the catcher tries to guess when the 
runner will steal. A pitchout every time will walk the batter, so there must be an optimal 
frequency—depending on the base runner and on the situation.) Again a small part of 
the game could be isolated and analyzed. 

On the other hand, blackjack is not a matrix game (in a casino) because the house 
follows fixed rules. My friend Ed Thorp found a winning strategy by counting high 
cards—forcing more shuffling and more decks at Las Vegas. There was no element of 
chance, and no mixed strategy x*. The best-seller Bringing Down the House tells how 
MIT students made a lot of money (while not doing their homework). 

There is also the Prisoner’s Dilemma, in which two accomplices are separately of- 
fered the same deal: Confess and you are free, provided your accomplice does not con- 
fess (the accomplice then gets 10 years). If both confess, each gets 6 years. If neither 
confesses, only a minor crime (2 years each) can be proved. What to do? The temptation 
to confess is very great, although if they could depend on each other they would hold 
out. This is not a zero-sum game; both can lose. 

One example of a matrix game is poker. Bluffing is essential, and to be effective it has 
to be unpredictable. (If your opponent finds a pattern, you lose.) The probabilities for 
and against bluffing will depend on the cards that are seen, and on the bets. In fact, the 
number of alternatives again makes it impractical to find an absolutely optimal strategy 
x*. A good poker player must come pretty close to x*, and we can compute it exactly if 
we accept the following enormous simplification of the game: 


X is dealt a jack or a king, with equal probability, and Y always gets a queen. X can 
fold and lose the $1 ante, or bet an additional $2. If X bets, Y can fold and lose $1, or 
match the extra $2 and see if X is bluffing. Then the higher card wins the $3 from the 
opponent. So Y has two possibilities, reacting to X (who has four strategies): 


Strategies (Row 1) If X bets, Y folds. 
for Y (Row 2) If X bets, Y matches the extra $2. 


(1) Bet the extra $2 on a king and fold on a jack. 
Strategies (2) Bet the extra $2 in either case (bluffing). 
for X (3) Fold in either case, and lose $1 (foolish). 
(4) Fold on a king and bet on a jack (foolish). 
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The payoff matrix A requires a little patience to compute: 


a}; =0: X loses $1 half the time on a jack and wins on a king (Y folds). 

a2, =1: Both bets X loses $1 half the time and wins $3 half the time. 
aj2=1: X bets and Y folds (the bluff succeeds). 

a}; =0: X wins $3 with the king and loses $3 with the jack (the bluff fails). 


Poker payoff matrix A= ; 
10-1 -2 


On te 4 | 


The optimal strategy for X is to bluff half the time, x* = (5, 5,0,0). The underdog Y 
must choose y* = (5,5). The value of the game is fifty cents to X. 

That is a strange way to end this book, by teaching you how to play watered-down 
poker (blackjack pays a lot better). But I guess even poker has its place within linear 
algebra and its applications. I hope you have enjoyed the book. 


Problem Set 8.5 


1. How will the optimal strategies in the game that opens this section be affected if the 
$20 is increased to $70? What is the value (the average win for X) of this new game? 


2. With payoff matrix A = [47], explain the calculation by X of the maximin and by Y 
of the minimax. What strategies x* and y* are optimal? 


3. If aj; is the largest entry in its row and the smallest in its column, why will X always 
choose column j and Y always choose row i (regardless of the rest of the matrix)? 
Show that the preceding problem had such an entry, and then construct an A without 
one. 


4. Compute Y’s best strategy by weighting the rows of A = [3 44] with y and 1—y. X 
will concentrate on the largest of the components 3y + 2(1—y), 4y, and y+ 3(1—y). 
Find the largest of those three (depending on y) and then find the y* between 0 and 1 
that makes this largest component as small as possible. 


5. With the same A as in Problem 4, find the best strategy for X. Show that X uses only 
the two columns (the first and third) that meet at the minimax point in the graph. 


6. Find both optimal strategies, and the value, if 


7. Suppose A = ie an What weights x; and 1 — x; will give a column of the form 


[u u]', and what weights y; and 1 — y; on the two rows will give a new row [v y]? 
Show that u = v. 
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8. 


10. 


11. 


12. 


13. 


14. 
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Find x*, y* and the value v for 


. Compute 


min max (x;yj +x2y2). 
rn ae (xiy1 +292) 


yy ty2=1 xj +%2=1 


Explain each of the inequalities in equation (5). Then, once the minimax theorem 
has turned them into equalities, derive (again in words) the saddle point equations 


(4). 


Show that x* = (5, 5,0,0) and y* = (5; ) are optimal strategies in our simplified 
version of poker, by computing yAx* and y*Ax and verifying the conditions (4) for a 
saddle point. 


Has it been proved that no chess strategy always wins for black? This is certainly 
true when the players are given two moves at a time; if black had a winning strategy, 
white could move a knight out and back and then follow that strategy, leading to the 
impossible conclusion that both would win. 


If X chooses a prime number and simultaneously Y guesses whether it is odd or even 
(with gain or loss of $1), who has the advantage? 


If X is a quarterback, with the choice of run or pass, and Y can defend against a run 


or a pass, suppose the payoff (in yards) is 


A= 
6 —6 


2 8 | defense against run 
defense against pass. 


run pass 


What are the optimal strategies and the average gain on each play? 


Intersection, Sum, and Product of Spaces 


A.1_ The Intersection of Two Vector Spaces 


New questions arise from considering two subspaces V and W, not just one. We look 
first at the vectors that belong to both subspaces. This “intersection” VM W is a subspace 
of those subspaces: 


If V and W are subspaces of one vector space, so is their intersection VW. 
The vectors belonging to both V and W form a subspace. 


Suppose x and y are vectors in V and also in W. Because V and W are vector spaces 
in their own right, x+y and cx are in V and in W. The results of addition and scalar 
multiplication stay within the intersection. 

Two planes through the origin (or two “hyperplanes” in R”) meet in a subspace. The 
intersection of several subspaces, or infinitely many, is again a subspace. 


Example 1. The intersection of two orthogonal subspaces V and W is the one-point 
subspace V™ W = {0}. Only the zero vector is orthogonal to itself. 


Example 2. Suppose V and W are the spaces of n by n upper and lower triangular 
matrices. The intersection VM W is the set of diagonal matrices—belonging to both 
triangular subspaces. Adding diagonal matrices, or multiplying by c, leaves a diagonal 
matrix. 


Example 3. Suppose V is the nullspace of A, and W is the null space of B. Then V1 W 
is the smaller nullspace of the larger matrix C: 

A 

RI 


Intersection of nullspaces N(A)MN(B) is the nullspace of C = 


Cx = 0 requires both Ax = 0 and Bx = 0. So x has to be in both nullspaces. 
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A.2. The Sum of Two Vector Spaces 


Usually, after discussing the intersection of two sets, it is natural to look at their Union. 
With vector spaces, this is not natural. The union V UW of two subspaces will not in 
general be a subspace. If V and W are the x-axis and the y-axis in the plane, the two 
axes together are not a subspace. The sum of (1,0) and (0, 1) is not on either axis. 

We do want to combine V and W. In place of their union we turn to their sum. 


Definition. If V and W are both subspaces of a given space, so is their sum. V+ W 
contains all combinations v+ w, where v is in V and w is in W. 


V + W is the smallest vector space that contains both V and W. The sum of the 
x-axis and the y-axis is the whole x-y plane. So is the sum of any two different lines, 
perpendicular or not. If V is the x-axis and W is the 45° line x = y, then any vector like 
(5,3) can be split into v-+ w = (2,0) + (3,3). Thus V+ W is all of R’. 


Example 4. Suppose V and W are orthogonal complements in R”. Then their sum is 
V+W=R". Every x is the sum of its projections in V and W. 


Example 5. If V is the space of upper triangular matrices, and W is the space of lower 
triangular matrices, then V + W is the space of all matrices. Every n by n matrix can be 
written as the sum of an upper and a lower triangular matrix—in many ways, because 
the diagonals are not uniquely determined. 

These triangular subspaces have dimension n(n+ 1) /2. The space V+ W of all matri- 
ces has dimension n*. The space VM W of diagonal matrices has dimension n. Formula 
(3) below becomes n* +n = n(n+1)/2+n(n+1)/2. 


Example 6. If V is the column space of A, and W is the column space of B, then V-+ W 
is the column space of the larger matrix |A B|. The dimension of V+ W may be less 
than the combined dimensions of V and W (because these two spaces might overlap): 
Sum of column spaces dim(V + W) = rank of [A Bj. (1) 
The computation of VW is more subtle. For the intersection of column spaces, a 
good method is to put bases for V and W in the columns of A and B. The nullspace 
of [A B] leads to VM W (see Problem 9). Those spaces have the same dimension (the 
nullity of [A B]). Combining with dim(V + W) gives 
dim(V + W) + dim(V NW) = rank of [A B] + nullity of [A B]. (2) 
We know that the rank plus the nullity (counting pivot columns plus free columns) al- 
ways equals the total number of columns. When [A B] has k+ ¢ columns, with k = dim V 
and ¢ = dimW, we reach a neat conclusion: 
Dimension formula dim(V + W) + dim(VN W) = dim(V)+dim(W). — (3) 
Not a bad formula. The overlap of V and W is in VM W. 
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A.3 The Cartesian Product of Two Vector Spaces 


If V has dimension n, and W has dimension gq, their Cartesian product V x W has di- 
mension n+ q. 


Definition. V <x W contains all pairs of vectors x = (v,w). 


Adding (v,w) to (v*,w*) in this product space gives (v-+v*,w+w*). Multiplying by c 
gives (cv,cw). All operations in V x W are a component at a time. 


Example 7. The Cartesian product of R? and R° is very much like R°. A typical vector 
x in R? x R? is ((1,2),(4,6,5)): one vector from R? and one from R°. That looks like 
(1,2,4,6,5) in R>. 


Cartesian products go naturally with block matrices. From R? to R°, we have ordinary 
5 by 5 matrices. On the product space R? x R°, the natural form of a matrix is a 5 by 5 
block matrix M: 
R’ to RR’ R? to R’ 
R’toR? R*toR? 


2by2 2by3 
3by2 3by3 


A B 
C D 


Matrix-vector multiplication produces (Av + Bw,Cv + Dw). Not too fascinating. 


A.4_ The Tensor Product of Two Vector Spaces 


Somehow we want a product space that has dimension n times gq. The vectors in this 
“tensor product” (denoted ©) will look like n by g matrices. For the tensor product 
R2 QR’, the vectors will look like 2 by 3 matrices. The dimension of R? x R? is 5, but 
the dimension of R? @ R? is going to be 6. 

Start with vy = (1,2) and w = (4,6,5) in R? and R°. The Cartesian product just puts 
them next to each other as (v,w). The tensor product combines v and w into the rank 1 


matrix vw': 


1 
Column times row v@w=ww! ; 4 6 5| = 


4 6 5 
8 12 10]. 


All the special matrices vw! belong to the tensor product R* @ R*. The product space 
is spanned by those vectors v®w. Combinations of rank-1 matrices give all 2 by 3 
matrices, so the dimension of R* @ R° is 6. Abstractly: The tensor product V & W is 
identified with the space of linear transformations from V to W. 

If V is only a line in R’, and W is only a line in R°, then V W is only a “line in 
matrix space.” The dimensions are now | x 1 = 1. All the rank-1 matrices vw! will be 
multiples of one matrix. 
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Basis for the Tensor Product. When V is R’ and W is R°, we have a standard 
basis for all 2 by 3 matrices (a six-dimensional space): 


1 0 0; JO 1 O} JO O 1] JO O OF} JO O OF JO O O 

0 0 0] {0 O OJ 0 0 OF }1 O OF JO 1 OF JO O 1]° 
That basis for R* @ R? was constructed in a natural way. I started with the standard 
basis vj = (1,0) and v2 = (0,1) for R*. Those were combined with the basis vectors 
w1 = (1,0,0), w2 = (0,1,0), and w3 = (0,0, 1) in R*. Each pair v; @ w; corresponds to 


one of the six basis vectors (2 by 3 matrices above) in the tensor product V ® W. This 
construction succeeds for subspaces too: 


Basis 


Basis: Suppose V and W are subspaces of R” and R? with bases v1,...,Vn 
and w1,...,W,. Then the ng rank-1 matrices vii are a basis for V & W. 


V & W is an ng-dimensional subspace of m by p matrices, An algebraist would match 
this matrix construction to the abstract definition of V@W. Then tensor products can 
go beyond the specific case of column vectors. 


A.5 The Kronecker Product A © B of Two Matrices 


An m by n matrix A transforms any vector v in R” to a vector Av in R”, Similarly, a p 
by g matrix B transforms w to Bw. The two matrices together transform vw! to Avw! BT. 
This is a linear transformation (of tensor products) and it must come from a matrix. 

What is the size of that matrix A ® B? It takes the nq-dimensional space R” ® R‘ to 
the mp-dimensional space R” ® R?. Therefore the matrix has shape mp by nq. We will 
write this Kronecker product (also called tensor product) as a block matrix: 


ayiB aj2B -:+ ainB 
Kronecker product B B .:.: B 
produ A@B= a2,;B a2 a2n (4) 
mp rows, ng columns . é ths ; 
AmB an2B --- AnnB 


Notice the special structure of this matrix! A lot of important block matrices have that 
Kronecker form. They often come from two-dimensional applications, where A is a 
“matrix in the x-direction” and B is acting in the y-direction (examples below). If A and 
B are square, som =n and p = q, then the big matrix A ® B is also square. 


Example 8. (Finite differences in the x and y directions) Laplace’s partial differen- 
tial equation —d*u/dx* — 0*u/dy" = 0 is replaced by finite differences, to find values 
for u on a two-dimensional grid. Differences in the x-direction add to differences in the 
y-direction, connecting five neighboring values of u: 
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o—_e—_-e —1 
isd, j + 2U4, 5 — Ui-1, j 
—1 2 -l 
ee 2 Ui, jt + 2U4, 5 — Ui, 5-1 
=0 
@—_e—__® —1 
x-differences y-differences sum 


A 5-point equation is centered at each of the nine meshpoints. The 9 by 9 matrix (call 
it Azp) is constructed from the 3 by 3 “1D” matrix for differences along a line: 


2 -1 O 1 O 
Difference matrix roe 1 Identity matrix t=lo1 
in one direction i eae in other direction - nie 


r- Oo SO 


Kronecker products produce three 1D differences along three lines, up or across: 


2 -!I O 
One direction A®lI=|-I 21 —-I 

QO —I 2] 

A 0 0 
Other direction I®A=|0 A O 

0 0A 

At+2i -I 0 

Both directions As =(A®@I)+(U@A)=]| -I A422 -I 


0 -[ A+2I 


The sum (A ®J) + (J @A) is the 9 by 9 matrix for Laplace’s five-point difference equation 
(Section 1.7 was for 1D and Section 7.4 mentioned 2D). The middle row of this 9 by 9 
matrix shows all five nonzeros from the five-point molecule: 


Away from boundary Row 5 of Azp = |0 =f. 2 4. HS -. 


Example 9. (The Fourier matrix in 2D) The one-dimensional Fourier matrix F is 
the most important complex matrix in the world. The Fast Fourier Transform in Section 
3.5 is a quick way to multiply by that matrix F. So the FFT transforms “time domain to 
frequency domain” for a 1D audio signal. For images we need the 2D transform: 


Transform along each row, 


Fourier matrix in 2D Inn =F @F= 
then down each column 


The image is a two-dimensional array of pixel values. It is transformed by Fp into a two- 
dimensional array of Fourier coefficients. That array can be compressed and transmitted 
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and stored. Then the inverse transform brings us back from Fourier coefficients to pixel 
values. We need to know the inverse rule for Kronecker products: 


The inverse of the matrix A ® B is the matrix A~! ® Bo. 


The FFT also speeds up the 2D inverse transform! We just invert in one direction fol- 
lowed by the other direction. We are adding YY cje!e!” over k and then £. 


The Laplace difference matrix A2zp = (A ®J) + (J @A) has no simple inverse formula. 
That is why the equation Aypu = b has been studied so carefully. One of the fastest 
methods is to diagonalize A2p by using its eigenvector matrix (which is the Fourier sine 
matrix S$ @S, very similar to Fyp). The eigenvalues of Azp come immediately from the 
eigenvalues of Ajp: 

The n? eigenvalues of (A @/) + (I @B) are all the sums A;(A) + A;(B). 
The n? eigenvalues of A ® B are all the products A;(A)A;(B). 
If A and B are n by n, the determinant of A ® B (the product of its eigenvalues) is 


(detA)"(detB)”. The trace of A @ B is (trace A)(trace B). This appendix illustrates both 
“pure linear algebra” and its crucial applications! 


Problem Set A 


1. Suppose S and T are subspaces of R'°, with dimS = 7 and dimT = 8. 


(a) What is the largest possible dimension of ST? 
(b) What is the smallest possible dimension of ST? 
(c) What is the smallest possible dimension of S+ T? 
(d) What is the largest possible dimension of S + T? 


2. What are the intersections of the following pairs of subspaces? 


(a) The x-y plane and the y-z plane in R°, 
(b) The line through (1,1, 1) and the plane through (1,0,0) and (0,1, 1). 
(c) The zero vector and the whole space R’. 
(d) The plane S perpendicular to (1, 1,0) and perpendicular to (0,1, 1) in R°. 
What are the sums of those pairs of subspaces? 
3. Within the space of all 4 by 4 matrices, let V be the subspace of tridiagonal matrices 
and W the subspace of upper triangular matrices. Describe the subspace V + W, 


whose members are the upper Hessenberg matrices. What is VM W? Verify formula 


(3). 


10. 
11. 
12. 


13. 
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If VM W contains only the zero vector, then equation (3) becomes dim(V + W) = 
dim V + dim W. Check this when V is the row space of A, W is the nullspace of A, 
and the matrix A is m by n of rank r. What are the dimensions? 


Give an example in R* for which VM W contains only the zero vector, but V is not 
orthogonal to W. 


If V1 W = {0}, then V + W is called the direct sum of V and W, with the special 
notation V @ W. If V is spanned by (1, 1,1) and (1,0,1), choose a subspace W so 
that V@ W = R°, Explain why any vector x in the direct sum V @ W can be written 
in one and only one way as x = v+w (with vin V and w in W). 


Find a basis for the sum V + W of the space V spanned by v; = (1,1,0,0), v2 = 
(1,0, 1,0) and the space W spanned by w, = (0,1,0,1), w2 = (0,0,1,1). Find also 
the dimension of VM W and a basis for it. 


Prove from equation (3) that rank(A + B) = rank(A) + rank(B). 


The intersection C(A)C(B) matches the nullspace of [A B]. Each y= Ax; = Bx2 
in the column spaces of both A and B matches x = (x1, —x2) in the nullspace, because 
[A Bl]x = Ax, — Bxz = 0. Check that y = (6,3,6) matches x = (1,1, —2, —3), and find 
the intersection C(A) 1C(B), for 


A 


NW we 
ROW 
& 

I 
SxS: bs 
Ne oO 


Multiply A @ B times A~! @ B™! to get AA~! @BB-! =1@1 = hp. 
What is the 4 by 4 Fourier matrix Fyp = F @F for F = |} |, |? 


Suppose Ax = A(A)x and By = A(B)y. Form a long column vector z with n* compo- 
nents, x,y, then x.y, and eventually x,y. Show that z is an eigenvector for (A @/)z = 
A(A)z and (A @B)z=A(A)A(B)z. 


What would be the seven-point Laplace matrix for —uy. — Uyy — Uzz = 0? This “three- 
dimensional” matrix is built from Kronecker products using J and Ajp. 


The Jordan Form 


Given a square matrix A, we want to choose M so that M~'AM is as nearly diagonal as 
possible. In the simplest case, A has a complete set of eigenvectors and they become 
the columns of M—otherwise known as S. The Jordan form is J = M~'AM = A; it is 
constructed entirely from 1 by 1 blocks J; = A;, and the goal of a diagonal matrix is 
completely achieved. In the more general and more difficult case, some eigenvectors are 
missing and a diagonal form is impossible. That case is now our main concern. 

We repeat the theorem that is to be proved: 


If a matrix A has s linearly independent eigenvectors, then it is similar to a 
matrix J that is in Jordan form, with s square blocks on the diagonal: 


Ji 
J—=M~'AM = 
Js 


Each block has one eigenvector, one eigenvalue, and is just above the diagonal: 


Ai 1 


An example of such a Jordan matrix is 


4 ae 
wf ll 


The double eigenvalue A = 8 has only a single eigenvector, in the first coordinate direc- 
tion e; = (1,0,0,0,0); as a result, A = 8 appears only in a single block J;. The triple 


WN 
| 
ooo o we 
ooo tm Re 
ooo oO Oo 
SO Or SO 
cae as E> |e > a > Fs >) 
| 
——1 
oOo Oo 
Oe = 
a 
| 
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eigenvalue A = 0 has two eigenvectors, e3 and es, which correspond to the two Jordan 
blocks Jz and J3. If A had 5 eigenvectors, all blocks would be 1 by 1 and J would be 
diagonal. 

The key question is this: JfA is some other 5 by 5 matrix, under what conditions will 
its Jordan form be this same J? When will there exist an M such that M~'AM = J? Asa 
first requirement, any similar matrix A must share the same eigenvalues 8, 8, 0, 0, 0. But 
the diagonal matrix with these eigenvalues is not similar to /—and our question really 
concerns the eigenvectors. 

To answer it, we rewrite M~'AM = J in the simpler form AM = MJ: 


8 1 
ANG Se Ke. Re KS SAP. he BS Ka XS 0 1 
0 0 
0 
Carrying out the multiplications a column at a time, 
Ax, = 8x and Ax? = 8x2 +X] (1) 
Ax3 = 0x3 and Ax4 = 0x4 +3 and Axs = Oxs. (2) 


Now we can recognize the conditions on A. It must have three genuine eigenvectors, 
just as J has. The one with A = 8 will go into the first column of M, exactly as it would 
have gone into the first column of S: Ax; = 8x,;, The other two, which will be named 
x3 and x5, go into the third and fifth columns of M: Ax3 = Axs = 0. Finally there must 
be two other special vectors, the generalized eigenvectors x2 and x4. We think of x2 as 
belonging to a string of vectors, headed by x; and described by equation (1). In fact, 
x2 1s the only other vector in the string, and the corresponding block J; is of order 2. 
Equation (2) describes two different strings, one in which x4 follows x3, and another in 
which xs is alone; the blocks J2 and J3 are 2 by 2 and 1 by 1. 

The search for the Jordan form of A becomes a search for these strings of vectors, 
each one headed by an eigenvector: For every i, 


either Ax; = Ajx; or Ax; = Ajxj + xj-1- (3) 


The vectors x; go into the columns of M, and each string produces a single block in J. 
Essentially, we have to show how these strings can be constructed for every matrix A. 
Then if the strings match the particular equations (1) and (2), our J will be the Jordan 
form of A. 

I think that Filippov’s idea makes the construction as clear and simple as possible!. 
It proceeds by mathematical induction, starting from the fact that every 1 by 1 matrix 


'A. F, Filippov, A short proof of the reduction to Jordan form, Moscow Univ. Math. Bull., volume 26 (1971) 
pp. 70-71. 
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is already in its Jordan form. We may assume that the construction is achieved for all 
matrices of order less than n—this is the “induction hypothesis”—and then explain the 
steps for a matrix of order n. There are three steps, and after a general description we 
apply them to a specific example. 


Step 1. If we assume A is singular, then its column space has dimension r < n. Looking 
only within this smaller space, the induction hypothesis guarantees that a Jordan 
form is possible—there must be r independent vectors w; in the column space 
such that 


either Aw; = Aw; or Aw; = Ajw; + Wi_-1- (4) 


Step 2. Suppose the nullspace and the column space of A have an intersection of dimen- 
sion p. Of course, every vector in the nullspace is an eigenvector corresponding 
to A =0. Therefore, there must have been p strings in step | that started from 
this eigenvalue, and we are interested in the vectors w; that come at the end of 
these strings. Each of these p vectors is in the column space, so each one is a 
combination of the columns of A: w; = Ay; for some y;. 


Step 3. The nullspace always has dimension n—r. Therefore, independent from its 
p-dimensional intersection with the column space, it must contain n —r— p 
additional basis vectors z; lying outside that intersection. 


Now we put these steps together to give Jordan’s theorem: 


The r vectors w;, the p vectors y;, and the n —r-— p vectors z; form Jordan 
strings for the matrix A, and these vectors are linearly independent. They go 
into the columns of M, and J = M~'AM is in Jordan form. 


If we want to renumber these vectors as x1,...,X,, and match them to equation (3), 
then each y; should be inserted immediately after the w; it came from; it completes a 
string in which A; = 0. The z’s come at the very end, each one alone in its own string; 
again the eigenvalue is zero, since the z’s lie in the nullspace. The blocks with nonzero 
eigenvalues are already finished at step 1, the blocks with zero eigenvalues grow by one 
row and column at step 2, and step 3 contributes any 1 by | blocks J; = [0]. 

Now we try an example, and to stay close to the previous pages we take the eigenval- 
ues to be 8, 8, 0, 0, 0: 


8 00 8 8 
000 8 8 
A=|00 000 
000 0 0 
000 0 8 


Step 1. The column space has dimension r = 3, and is spanned by the coordinate vectors 
€1,€2,e5. To look within this space we ignore the third and fourth rows and 
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columns of A; what is left has eigenvalues 8, 8, 0, and its Jordan form comes 
from the vectors 


8 0 0 
0 1 8 
Wi = LO} 5 w2= |01, w3= |0 
0 0 0 
0 1 0 


The w; are in the column space, they complete the string for A = 8, and they 
start the string for A = 0: 


Aw, = 8w1, Aw2 = 8w2+w1, Aw3 = 0w3. (5) 
Step 2. The nullspace of A contains e2 and e3, so its intersection with the column space 
is spanned by e2. Therefore p = 1 and, as expected, there is one string in equa- 


tion (3) corresponding to A = 0. The vector w3 comes at the end (as well as the 
beginning) of that string, and w3 = A(e4 — e;). Therefore y = e4 — e}. 


Step 3. The example has n —r— p=5—3—1=1, and z = e3 is in the nullspace but 
outside the column space. It will be this z that produces a 1 by 1| block in J. 


If we assemble all five vectors, the full strings are 
Aw; =8w,, Aw2=8w2+w1, Aw3=0w3, Ay=Oy+w3, Az= Oz. 


Comparing with equations (1) and (2), we have a perfect match—the Jordan form of our 
example will be exactly the J we wrote earlier. Putting the five vectors into the columns 
of M must give AM = MJ, or M~'AM = J: 


8 0 0 —1 O 
018 0 0 
M=!0 00 0 1 
000 1 O 
010 0 0 


We are sufficiently trustful of mathematics (or sufficiently lazy) not to multiply out 
M~'AM. 

In Filippov’s construction, the only technical point is to verify the independence of 
the whole collection w;, y;, and z;. Therefore, we assume that some combination is zero: 


Yewit Vi doit Y gizi =0. (6) 
Multiplying by A, and using equations (4) for the w; as well as Az; = 0, 
Ai; 


ee or + y dAy; = 0. (7) 
Aiwitwi-t 
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The Ay; are the special w; at the ends of strings corresponding to A; = 0, so they cannot 
appear in the first sum. (They are multiplied by zero in A;w;.) Since equation (7) is 
some combination of the w;, which were independent by the induction hypothesis—they 
supplied the Jordan form within the column space—we conclude that each d; must be 
zero. Returning to equation (6), this leaves )’c;jw; = — ¥ g;z;, and the left-hand side is in 
the column space. Since the z’s were independent of that space, each g; must be zero. 
Finally, )' cjw; = 0, and the independence of the w; produces c; = 0. 

If the original A had not been singular, the three steps would have been applied instead 
to A’ = A —cl. (The constant c is chosen to make A’ singular, and it can be any one of the 
eigenvalues of A.) The algorithm puts A’ into its Jordan form M~!A’M = J' by producing 
the strings x; from the w;, y; and z;. Then the Jordan form for A uses the same strings 
and the same M: 


M-'AM = M7 !A'M+M7'!ceM =J' +cI = J. 


This completes the proof that every A is similar to some Jordan matrix J. Except for a 
reordering of the blocks, it is similar to only one such J; there is a unique Jordan form 
for A. Thus, the set of all matrices is split into a number of families, with the following 
property: All the matrices in the same family have the same Jordan form, and they are 
all similar to each other (and to J), but no matrices in different families are similar. In 
every family, J is the most beautiful—if you like matrices to be nearly diagonal. With 
this classification into families, we stop. 


Example 1. 
0 1 2 
A=1]0 0 1 with A =0,0,0. 
00 0 
This matrix has rank r = 2 and only one eigenvector. Within the column space, there is 
a single string w1, w2, which happens to coincide with the last two columns: 


or 
Aw, =0 and Aw2 = Ow2+ wy]. 


The nullspace lies entirely within the column space, and it is spanned by w,. Therefore 
p = | in step 2, and the vector y comes from the equation 


2, 0 
Ay=w2= |1], where solutionis y= |0 
0 1 
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Finally, the string w1, w2, y goes into the matrix M: 


0 
Ol, and M~'AM = |0 
1 0 


SS 

S-HaS 
I 
~ 


Application to du/dt = Au 


As always, we simplify the problem by uncoupling the unknowns. This uncoupling is 
complete only when there is a full set of eigenvectors, and u = Sv; the best change of 
variables in the present case is u = Mv. This produces the new equation Mdv/dt = AMv, 
or dv/dt = Jv, which is as simple as the circumstances allow. It is coupled only by the 
off-diagonal 1s within each Jordan block. In the preceding example, which has a single 
block, du/dt = Au becomes 


dv 0 1 0 da/dt =b a = ao + bot + cot? /2 
= 00 1|v or db/dt =c or = bo + cot 
00 0 dc/dt =0 c= Co. 


The system is solved by working upward from the last equation, and a new power of t 
enters at every step. (An £ by £ block has powers as high as t’~!.) The exponentials of 
J, in this case and in the earlier 5 by 5 example, are 


e* te! 0 0 0 

1 #9772 0: 2 00.0 
e'=|0 1 #¢ and 0 0 1t oO 
00 1 0 0010 

O° 0 -0F: 0.4 


You can see how the coefficients of a, b, and c appear in the first exponential. And in the 
second example, you can identify all five of the “special solutions” to du/dt = Au. Three 


Bt, u3 = et 


of them are the pure exponentials u; = e x3, and us = e"x5, formed as usual 
from the three eigenvectors of A. The other two involve the generalized eigenvectors x2 
and x4: 


uy = e* (tx, +. x2) and = ug = € (tx3 +.x4). (8) 


The most general solution to du/dt = Au is a combination cju, + ----+cs5us, and the 
combination that matches uo at time t = 0 is again 


Up = CyXy +°-++05%5, or uo = Mc, or c=M~'upo. 


This only means that u = Me’’M~!uo, and that the S and A in the old formula Se S~!ug 
have been replaced by M and J. 
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Problem Set B 


1. Find the Jordan forms (in three steps!) of 


1 1 
1 1 


01 2 
A= | and B= 10 0 0 
0 0 0 


2. Show that the special solution uz in equation (17) does satisfy du/dt = Au, exactly 
because of the string Ax; = 8x;, Ax7 = 8x7 +1. 


3. For the matrix B in Problem 1, use Me/’M~—! to compute the exponential e’, and 
compare it with the power series J + Bt + (Bt)?/2!+---. 


4. Show that each Jordan block J; is similar to its transpose, J} = P~'J;P, using the 
permutation matrix P with Is along the cross-diagonal (lower left to upper right). 
Deduce that every matrix is similar to its transpose. 


5. Find “by inspection” the Jordan forms of 
1-23 
1 1 
A=1]0 4 5 and =|! : 
0 0 6 


6. Find the Jordan form J and the matrix M for A and B (B has eigenvalues 1, 1, 1, —1). 
What is the solution to du/dt = Au, and what is e4’? 


00100 
00010 Poe 
Oe Br Or oF 
A=!]0 0001 and a eee i 4 
00000 
2 -1 2 0 
00000 


7. Suppose that A? = A. Show that its Jordan form J = M~!AM satisfies J? = J. Since 
the diagonal blocks stay separate, this means J? = J; for each block; show by direct 
computation that J; can only be a 1 by 1 block, J; = [0] or J; = [1]. Thus, A is similar 
to a diagonal matrix of Os and Is. 


Note. This is a typical case of our closing theorem: The matrix A can be diagonalized if 
and only if the product (A —A,I)(A—Agl)-+-(A—Apl), without including any repetitions 
of the 2’s, is zero. One extreme case is a matrix with distinct eigenvalues; the Cayley- 
Hamilton theorem says that with n factors A — AJ we always get zero. The other extreme 
is the identity matrix, also diagonalizable (p = 1 and A —J = 0). The nondiagonalizable 
matrix A = [41] satisfies not (A — 7) = 0 but only (A —/)? = 0—an equation with a 
repeated root. 


Matrix Factorizations 


L A=lU = lower triangular L upper triangular U 
a ~ \ 1s on the diagonal / \ pivots on the diagonal 


Requirements: No row exchanges as Gaussian elimination reduces A to U. 


lower triangular ;) ( pivot matrix Gs triangular U 


2. A=LDU = ‘ pee : 
1s on the diagonal / \ D is diagonal 1s on the diagonal 


Requirements: No row exchanges. The pivots in D are divided out to leave Is in 
U. If A is symmetric, then U is L' and A = LDL!'. 


3. PA=LU (permutation matrix P to avoid zeros in the pivot positions). 


Requirements: A is invertible. Then P, L, U are invertible. P does the row ex- 
changes in advance. Alternative: A = L)P,\U}. 


4. EA=R (mx mi invertible £) (any A) = rref(A). 

Requirements: None! The reduced row echelon form R has r pivot rows and pivot 
columns. The only nonzero in a pivot column is the unit pivot. The Last m — r rows 
of E are a basis for the left nullspace of A. and the first r columns of E~! are a basis 
for the column space of A. 

5. ASCO = (lower triangular matrix Cc) (transpose is upper triangular 
Requirements: A is symmetric and positive definite (all n pivots in D are positive). 
This Cholesky factorization has C = Ly’D. 

6. A=QR= (orthonormal columns in 0) (upper triangular R) 

Requirements: A has independent columns. Those are orthogonalized in Q by the 
Gram-Schmidt process. If A is square, then Q~'! = Q?. 
1. BSSKS-' = (cigenvectors in s) (cigenvalues in A) (lett eigenvectors in S =) 


Requirements: A must have n linearly independent eigenvectors. 
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8. 


10. 


11. 


12. 


13. 


14. 


15. 


Appendix C Matrix Factorizations 


A=QAQ'= (orthogonal matrix Q) (real eigenvalue matrix A) fey is o-') : 


Requirements: A is symmetric. This is the Spectral Theorem. 


. A=MJM~! = ( generalized eigenvectors in M ) (3 ordan blocks in J ) (m - ®) 


Requirements: A is any square matrix. Jordan form J has a block for each inde- 
pendent eigenvector of A. Each block has one eigenvalue. 


A=usvt = orthogonal m Xn matrix & orthogonal 
7 ~ \Uismxm) \oy,...,6, ondiagonal] \ Visnxn }” 
Requirements: None. This singular value decomposition (SVD) has the eigenvec- 


tors of AA™ in U and of ATA in V; 6; = \/A,(ATA) = \/A;(AA?). 


At=vetyul = orthogonal \ / diagonal n x m \ /[ orthogonal . 

nxn 1/o1,...,1/0, mxm 
Requirements: None. The pseudoinverse has ATA = projection onto row space of 
A and AA* = projection onto column space. The shortest least-squares solution to 
Ax = bis*=A*b. This solves A'AX = Alb. 
A=QH = (orthogonal matrix Q) (symmetric positive definite matrix H ) 


Requirements: A is invertible. This polar decomposition has H* = A'A. The 
factor H is semidefinite if A is singular. The reverse polar decomposition A = KQ 
has K? = AA’. Both have Q = UV! from the SVD. 


A=UAU~!= (unitary U ) (cigenvalue matrix A) (u robes (ee u") ; 


Requirements: A is normal: ANA = AA", Its orthonormal (and possibly complex) 
eigenvectors are the columns of U. Complex A’s unless A = A. 


A=U0TU = (unitary u) (triangular T with A’s on diagonal (eae = u") 


Requirements: Schur triangularization of any square A. There is a matrix U with 
orthonormal columns that makes U~!AU triangular. 


I D 
I —D 


F,= Fi, /2 even-odd 


| = one step of the FFT. 


F,/2| | permutation 


Requirements: F, = Fourier matrix with entries w/* where w”? = 1, w = e27//”, 
Then F,F, = nl. D has 1,w,w*,... on its diagonal. For n = 2° the Fast Fourier 
Transform has snl multiplications from @ stages of D’s. 
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Adjacency matrix of a graph Square matrix with a;; = 1 when there is an edge from 
node i to node j; otherwise a;; =0. A = A! for an undirected graph. 


Affine transformation 7 (v) = Av + vo = linear transformation plus shift. 
Associative Law (AB)C = A(BC) Parentheses can be removed to leave ABC. 


Augmented matrix [A b| Ax = b is solvable when b is in the column space of A; then 
[A b] has the same rank as A. Elimination on [A b] keeps equations correct. 


Back substitution Upper triangular systems are solved in reverse order x, to x1. 


Basis for V_ Independent vectors v1,...,vg whose linear combinations give every v in 
V. A vector space has many bases! 


Big formula for 1 by n determinants det(A) is a sum of 1! terms, one term for each 
permutation P of the columns. That term is the product aq ---dyq@ down the diagonal 
of the reordered matrix, times det(P) = +1. 


Block matrix A matrix can be partitioned into matrix blocks, by cuts between rows 
and/or between columns. 


Block multiplication of AB is allowed if the block shapes permit (the columns of A 
and rows of B must be in matching blocks). 


Cayley-Hamilton Theorem p(/) = det(A— AJ) has p(A) = zero matrix. 


Change of basis matrix M_ The old basis vectors v; are combinations )' m;;w; of the 
new basis vectors. The coordinates of cj vj +--+ +CyVy = djw, +---+dywy are related 
by d = Mc. (For n = 2, set vj = mw, +m21W2, V2 = Mj2wW1 +™22W2.) 


Characteristic equation det(A —AJ) =0 Then roots are the eigenvalues of A. 


Cholesky factorization A —=CC! = (LVD)(LVD)' for positive definite A. 
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Circulant matrix C Constant diagonals wrap around as in cyclic shift $. Every 
circulant is col +c)S+-::+c,_1S"” '. Cx = convolution c * x. Eigenvectors in F. 


Cofactor C;; Remove row i and column j; multiply the determinant by (—1)!*/. 


Column picture of Ax =b The vector b becomes a combination of the columns of A. 
The system is solvable only when b is in the column space C(A). 


Column space C(A) Space of all combinations of the columns of A. 
Commuting matrices AB = BA If diagonalizable, they share n eigenvectors. 


Companion matrix Put cj,...,c, inrow n and put n— | 1s along diagonal |. Then 
det(A — AI) =+(cy +024 +0347 +--+). 


Complete solution x =x,+x, toAx=b (Particular x,) + (Xp in nullspace). 
Complex conjugate z= a — ib for any complex number z = a + ib.Then zz = |z|?. 


Condition number cond(A) = K(A) = ||A||||A7!|| = Omax/Omin In Ax = b, the 
relative change ||6x||/||x|| is less than cond(A) times the relative change ||6b||/||b]]. 
Condition numbers measure the sensitivity of the output to change in the input. 


Conjugate Gradient Method A sequence of steps to solve positive definite Ax = b 
by minimizing 5xTAx —x"b over growing Krylov subspaces. 


Covariance matrix © When random variables x; have mean = average value = 0, 
their covariances %;; are the averages of x;x;. With means X;, the matrix © = mean of 
(x —x)(x —x)' is positive (semi)definite; it is diagonal if the x; are independent. 


Cramer’s Rule for Ax=b_B; has b replacing column j of A, and x; = |B;|/|Al. 


Cross product u x vin R° Vector perpendicular to u and y, length ||u||||v||| sin @| = 
parallelogram area, computed as the “determinant” of [i j k; uy uz u33 V1 v2 V3}. 


Cyclic shift S Permutation with sz; = 1, 537 = 1,..., finally s,, = 1. Its eigenvalues 


are nth roots e2”"*/" of 1; eigenvectors are columns of the Fourier matrix F’. 


Determinant |A| = det(A) Defined by det/ = 1, sign reversal for row exchange, and 
linearity in each row. Then |A| = 0 when A is singular. Also |AB| = |A||BI, 

|A~!| = 1/|A], and |A™| = |A]. The big formula for det(A) has a sum of n! terms, the 
cofactor formula uses determinants of size n — 1, volume of box = | det(A)}. 


Diagonal matrix D dj; = 0 if i 4 j. Block-diagonal: zero outside square blocks Dj. 


Diagonalizable matrix A Must have n independent eigenvectors (in the columns of 
S; automatic with n different eigenvalues). Then S~'AS = A = eigenvalue matrix. 
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Diagonalization A = S~'AS__A = eigenvalue matrix and S = eigenvector matrix. A 
must have n independent eigenvectors to make S invertible. All A‘ = SAKS~!. 


Dimension of vector space dim(V) = number of vectors in any basis for V. 
Distributive Law A(B+C)=AB-+AC_ Add then multiply, or multiply then add. 


Dot product x'y = x,y; +---+2%ny, Complex dot product is xy. Perpendicular 
vectors have zero dot product. (AB);; = (row i of A) - (column j of B). 


Echelon matrix U The first nonzero entry (the pivot) in each row comes after the 
pivot in the previous row. All zero rows come last. 


Eigenvalue J and eigenvector x Ax = Ax with x £0, so det(A — AJ) = 0. 
Eigshow Graphical 2 by 2 eigenvalues and singular values (MATLAB or Java). 


Elimination A sequence of row operations that reduces A to an upper triangular U or 
to the reduced form R = rref(A). Then A = LU with multipliers @;; in L, or PA = LU 
with row exchanges in P, or EA = R with an invertible E. 


Elimination matrix = Elementary matrix E;; The identity matrix with an extra 
—€;; in the i, j entry (i 4 j). Then £;;A subtracts ¢;; times row j of A from row i. 


Ellipse (or ellipsoid) x'Ax = 1 A must be positive definite; the axes of the ellipse are 
eigenvectors of A, with lengths 1/2. (For ||x|| = 1 the vectors y = Ax lie on the 
ellipse ||A~!y||? = y'(AA')~!y = 1 displayed by eigshow; axis lengths 6;.) 


Exponential e“’ = 7+ Ar-+(Art)*/2!+--- has derivative Ae“; e4’u(0) solves u! = Au. 


Factorization A = LU If elimination takes A to U without row exchanges, then the 
lower triangular L with multipliers ¢;; (and ¢;; = 1) brings U back to A. 


Fast Fourier Transform (FFT) A factorization of the Fourier matrix F;, into 
£ = log, n matrices S; times a permutation. Each S; needs only n/2 multiplications, so 
F,x and F-'c can be computed with n@/2 multiplications. Revolutionary. 


Fibonacci numbers 0, 1, 1, 2, 3, 5,... satisfy 

Fy = Fy-1 + Fr-2 = (Af — 43) /(An — Az). Growth rate Ay = (1+ V5) /2 the largest 
eigenvalue of the Fibonacci matrix || }]. 

Four fundamental subspaces of A C(A), N(A), C(A!), N(A!). 

Fourier matrix F Entries Fi, = emkin give orthogonal columns FF =nl. Then 
y = Fc is the (inverse) Discrete Fourier Transform y; = vr cpe2tsk/ as 


Free columns of A Columns without pivots; combinations of earlier columns. 
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Free variable x; Column i has no pivot in elimination. We can give the n — r free 
variables any values, then Ax = b determines the r pivot variables (if solvable!). 


Full column rank r=n_ Independent columns, N(A) = {0}, no free variables. 


Full row rank r=m _ Independent rows, at least one solution to Ax = b, column space 
is all of R”. Full rank means full column rank or full row rank. 


Fundamental Theorem The nullspace N(A) and row space C(A!) are orthogonal 
complements (perpendicular subspaces of R” with dimensions r and n — r) from 

Ax = 0. Applied to A', the column space C(A) is the orthogonal complement of 
N(A?). 


Gauss-Jordan method Invert A by row operations on [A J] to reach [J A~']. 


Gram-Schmidt orthogonalization A = QR_ Independent columns in A, orthonormal 
columns in Q. Each column q; of Q is a combination of the first 7 columns of A (and 
conversely, so R is upper triangular). Convention: diag(R) > 0. 


Graph G Set of m nodes connected pairwise by m edges. A complete graph has all 
n(n — 1)/2 edges between nodes. A tree has only n— | edges and no closed loops. A 
directed graph has a direction arrow specified on each edge. 


Hankel matrix H Constant along each antidiagonal; h;; depends on i+ j. 

aad , =T : eee 
Hermitian matrix A! =A =A Complex analog of a symmetric matrix: @ fi = Gij- 
Hessenberg matrix H Triangular matrix with one extra nonzero adjacent diagonal. 


Hilbert matrix hilb(n) Entries Hj; = 1/(i+ j—1) = Jy x’ !x/—!dx. Positive definite 
but extremely small Ain and large condition number. 


Hypercube matrix P? Row n+ | counts corners, edges, faces, ..., of a cube in R". 
Identity matrix / (or [,) Diagonal entries = 1, off-diagonal entries = 0. 


Incidence matrix of a directed graph The m by n edge-node incidence matrix has a 
row for each edge (node i to node /), with entries —1 and | in columns i and /. 


Indefinite matrix A symmetric matrix with eigenvalues of both signs (+ and —). 


Independent vectors v,,...,v, No combination cjvj +---+c,vg = zero vector 
unless all c; = 0. If the v’s are the columns of A, the only solution to Ax = 0 is x = 0. 


Inverse matrix A~! Square matrix with A~'A = J and AA~! = J. No inverse if 
detA = 0 and rank(A) <n, and Ax = 0 for a nonzero vector x. The inverses of AB and 
A’ are B-'A~! and (A~!) Cofactor formula (A~!);; = Cji/detA. 


Appendix D Glossary: A Dictionary for Linear Algebra 479 


Iterative method A sequence of steps intended to approach the desired solution. 


Jordan form J = M~'AM_ If A has s independent eigenvectors, its “generalized” 
eigenvector matrix M gives J = diag(J,,...,Js). The block Jy is Agd; + Nx where N; has 
1s on diagonal 1. Each block has one eigenvalue A; and one eigenvector (1,0,...,0). 


Kirchhoff’s Laws Current law: net current (in minus out) is zero at each node. 
Voltage law: Potential differences (voltage drops) add to zero around any closed loop. 


Kronecker product (tensor product) A@B_ Blocks a;;B, eigenvalues 1,,(A)Aq(B). 


Krylov subspace K;(A,b) The subspace spanned by b, Ab,...,A/~!b. Numerical 
methods approximate A~!b by x j with residual b — Ax; in this subspace. A good basis 
for K; requires only multiplication by A at each step. 


Least-squares solution The vector x that minimizes the error ||e||* solves 
A'Ax = A'b. Then e = b — AX is orthogonal to all columns of A. 


Left inverse A* If A has full column rank n, then At = (ATA)~!AT has ATA = Ih. 
Left nullspace N(A!) Nullspace of A‘ = “left nullspace” of A because y'A = 01. 
Length ||x|| Square root of x'x (Pythagoras in n dimensions). 

Linear combination cv + dw or )icjv; Vector addition and scalar multiplication. 


Linear transformation T Each vector v in the input space transforms to T(v) in the 
output space, and linearity requires T(cv-+ dw) = cT(v) +dT(w). Examples: Matrix 
multiplication Av, differentiation in function space. 


Linearly dependent v,,...,v,, A combination other than all c; = 0 gives )'cjv; = 0. 


Lucas numbers L=2,1,3,4,..., satisfy Ly = Ly—1 + Ln-2 = A+ A7, with 
eigenvalues A),A2 = (1+ V5)/2 of the Fibonacci matrix | | }]. Compare Lo = 2 with 
Fibonacci. 


Markov matrix M_ All m;; > 0 and each column sum is 1. Largest eigenvalue A = 1. 
If m;; > 0, the columns of M k approach the steady-state eigenvector Ms = s > 0. 


Matrix multiplication AB The i, j entry of AB is (row i of A) - (column j of B) 

= Vajxb;;. By columns: column j of AB = A times column j of B. By rows: row i of A 
multiplies B. Columns times rows: AB = sum of (column k)(row k). All these 
equivalent definitions come from the rule that AB times x equals A times Bx. 


Minimal polynomial of A The lowest-degree polynomial with m(A) = zero matrix. 
The roots of m are eigenvalues, and m(A) divides det(A — AJ). 


Multiplication Ax = x;(column 1) +---+.x,(column n) = combination of columns. 
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Multiplicities AM and GM The algebraic multiplicity AM of an eigenvalue A is the 
number of times A appears as a root of det(A — AJ) = 0. The geometric multiplicity 
GM is the number of independent eigenvectors (= dimension of the eigenspace for A). 


Multiplier ¢;; The pivot row j is multiplied by ¢;; and subtracted from row i to 
eliminate the i, j entry: ¢;; = (entry to eliminate) /(jth pivot). 


Network A directed graph that has constants c1,...,Cm associated with the edges. 


Nilpotent matrix N Some power of N is the zero matrix, N‘ = 0. The only 
eigenvalue is A = 0 (repeated n times). Examples: triangular matrices with zero 
diagonal. 


Norm ||A|| of a matrix The “¢? norm” is the maximum ratio ||Ax||/||x|| = Omax. Then 
||Ax|] < ||A]]||l], [ABI < |[Al|||B]|, and ||A + Bl] < |/A]| + |[B]|. Frobenius norm 
|Al|% = LYaj;; ¢! and ¢* norms are largest column and row sums of |q;/|. 


Normal equation A'Ax = A'b Gives the least-squares solution to Ax = b if A has full 
rank n. The equation says that (columns of A) - (b— Ax) = 0. 


Normal matrix VN NN! = N'N, leads to orthonormal (complex) eigenvectors. 
Nullspace matrix N The columns of N are the n — r special solutions to As = 0. 
Nullspace N(A) Solutions to Ax = 0. Dimension n — r = (# columns) — rank. 


Orthogonal matrix Q Square matrix with orthonormal columns, so Q'Q = J implies 
Q' = Q"!. Preserves length and angles, ||Qx|| = ||x|| and (Qx)'(Qy) =x'y. All 
|A| = 1, with orthogonal eigenvectors. Examples: Rotation, reflection, permutation. 


Orthogonal subspaces Every v in V is orthogonal to every w in W. 


Orthonormal vectors ¢1,...,¢, Dot products are g!q; = 0, if i¢ j and q/q; =1. 
The matrix Q with these orthonormal columns has Q'Q = /. If m=n, then Q' = Qu! 
and q1,.--,n is an orthonormal basis for R": every v = Y-(v"q;)q;- 


T 


Outer product is wv" column times row = rank-1 matrix. 


Partial pivoting In elimination, the jth pivot is chosen as the largest available entry 
(in absolute value) in column j. Then all multipliers have |¢;;| < 1. Roundoff error is 
controlled (depending on the condition number of A). 


Particular solution x, Any solution to Ax = b; often x, has free variables = 0. 


Pascal matrix Ps = pascal(n) The symmetric matrix with binomial entries ga mae 
Ps; = P,Py all contain Pascal’s triangle with det = 1 (see index for more properties). 
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Permutation matrix P There are n! orders of 1,...,7; the n! P’s have the rows of J in 
those orders. PA puts the rows of A in the same order. P is a product of row exchanges 
P,;; P is even or odd (det P = 1 or —1) based on the number of exchanges. 


Pivot columns of A Columns that contain pivots after row reduction; not 
combinations of earlier columns. The pivot columns are a basis for the column space. 


Pivot d_ The first nonzero entry when a row is used in elimination. 


Plane (or hyperplane) in R"” Solutions to a'x = 0 give the plane (dimension n — 1) 
perpendicular to a £ 0. 


Polar decomposition A = QH Orthogonal Q, positive (semi)definite H. 


Positive definite matrix A Symmetric matrix with positive eigenvalues and positive 
pivots. Definition: x'Ax > 0 unless x = 0. 


Projection matrix P onto subspace S_ Projection p = Pb is the closest point to b in 
S, error e = b— Pb is perpendicular to S. P* = P = P', eigenvalues are | or 0, 
eigenvectors are in S or S~. If columns of A = basis for S, then P= A(ATA)~!AT. 


Projection p = a(a'b/a'a) onto the line through a P = aa! /a‘a has rank 1. 


Pseudoinverse A* (Moore-Penrose inverse) Then by m matrix that “inverts” A 
from column space back to row space, with N(At+) = N(A'). A*A and AA® are the 
projection matrices onto the row space and column space. rank(A*) = rank(A). 


Random matrix rand(n) or randn(n) MATLAB creates a matrix with random entries, 
uniformly distributed on [0 1] for rand, and standard normal distribution for randn. 


Rank 1 matrix A = uv' 40 Column and row spaces = lines cu and cv. 


Rank 7(A) Equals number of pivots = dimension of column space = dimension of 
row space. 


Rayleigh quotient g(x) =x'Ax/x'x For A =A!, Amin < q(x) < Amax. Those 


extremes are reached at the eigenvectors x for Amin(A) and Amax (A). 


Reduced row echelon form R = rref(A) Pivots= 1; zeros above and below pivots; r 
nonzero rows of R give a basis for the row space of A. 


Reflection matrix Q = 7—2uu' The unit vector u is reflected to Qu = —u. All 
vectors x in the plane u'x = 0 are unchanged because Qx = x. The “Householder 
matrix” has Q' = Q-!=@Q. 


Right inverse A* If A has full row rank m, then At = A'(AA™)~! has AAt = Ip. 
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Rotation matrix R= hee 7 te | rotates the plane by 0, and R~! = R? rotates back 


sinO cos@ : : 
by —@. Orthogonal matrix, eigenvalues e’? and e~’®, eigenvectors (1, +i). 


Row picture of Ax = b Each equation gives a plane in R” planes intersect at x. 
Row space C(A!) All combinations of rows of A. Column vectors by convention. 


Saddle point of f(x;,...,x,) A point where the first derivatives of f are zero and the 
second derivative matrix (07 f /Ox;0x ; = Hessian matrix) is indefinite. 


Schur complement S = D—CA~'B Appears in block elimination on [4 3]. 
Schwarz inequality |v-w| < ||v||||w|| Then |v'Aw|? < (vTAv)(w! Aw) if A =C'C. 


Semidefinite matrix A (Positive) semidefinite means symmetric with x'Ax > 0 for 
all vectors x. Then all eigenvalues A > 0; no negative pivots. 


Similar matrices A and B_ B= M~'AM has the same eigenvalues as A. 


Simplex method for linear programming The minimum cost vector x* is found by 
moving from corner to lower-cost corner along the edges of the feasible set (where the 
constraints Ax = b and x > 0 are satisfied). Minimum cost at a corner! 


Singular matrix A A square matrix that has no inverse: det(A) = 0. 


Singular Value Decomposition (SVD) A = U rl= (orthogonal U) times (diagonal 
Y) times (orthogonal V') First r columns of U and V are orthonormal bases of C’(A) 
and C(A!), with Av; = oju; and singular value 0; > 0. Last columns of U and V are 
orthonormal bases of the nullspaces of AT and A. 


Skew-symmetric matrix K The transpose is —K, since K;; = —Kj;. Eigenvalues are 
pure imaginary, eigenvectors are orthogonal, e*’ is an orthogonal matrix. 


Solvable system Ax = b_ The right side b is in the column space of A. 
Spanning set v|,...,¥j, for V Every vector in V is a combination of v1,...,Vm. 
Special solutions to As =0 One free variable is s; = 1, other free variables = 0. 


Spectral theorem A = QAQ' Real symmetric A has real A; and orthonormal g;, with 
Aq; = Aiqi. In mechanics, the g; give the principal axes. 


Spectrum of A The set of eigenvalues {A1,..., A}. Spectral radius = |Amax|. 
Standard basis for R’ Columns of n by n identity matrix (written i, j,k in R°). 


Stiffness matrix K When x gives the movements of the nodes in a discrete structure, 
Kx gives the internal forces. Often K = A'CA, where C contains spring constants from 
Hooke’s Law and Ax = stretching (strains) from the movements x. 
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Subspace S of V_ Any vector space inside V, including V and Z = {zero vector}. 


Sum V+ W of subspaces Space of all (v in V)+(w in W). Direct sum: 
dim(V + W) = dim V + dim W, when V and W share only the zero vector. 


Symmetric factorizations A = LDL! and A = QAQ' The number of positive pivots 
in D and positive eigenvalues in A is the same. 


Symmetric matrix A The transpose is A! =A, and a; j = Aji. A! is also symmetric. 
All matrices of the form R'R and LDL" and QAQ! are symmetric. Symmetric matrices 
have real eigenvalues in A and orthonormal eigenvectors in Q. 


Toeplitz matrix T Constant-diagonal matrix, so t;; depends only on j —i. Toeplitz 
matrices represent linear time-invariant filters in signal processing. 


Trace of A Sum of diagonal entries = sum of eigenvalues of A. TrAB = TrBA. 


Transpose matrix A! Entries Aij = Ajj. A’ isn by m, A'A is square, symmetric, 
positive semidefinite. The transposes of AB and A~! are B'A! and (A!)~!. 


Triangle inequality ||u + v|| < ||u||+ ||v|| For matrix norms, 


A+B|| < ||Al| + [BI 
Tridiagonal matrix T  ¢;; = Oif |i— j| > 1. T~' has rank 1 above and below diagonal. 
Unitary matrix U# = U' =U~-! Orthonormal columns (complex analog of Q). 


Vandermonde matrix V_ Vc = b gives the polynomial p(x) = co +--+ +en_1x"" | 
with p(x;) = 5; at n points. V;; = (x;)/~!, and detV = product of (x, —x;) for k >i. 


Vector addition v+w= (vj4+W},...,V¥n-+Wn») = diagonal of parallelogram. 


Vector space V_ Set of vectors such that all combinations cv + dw remain in V. Eight 
required rules are given in Section 2.1 for cv+ dw. 


Vector vin R” Sequence of n real numbers v = (1j,...,¥,) = point in R". 
Volume of box The rows (or columns) of A generate a box with volume | det(A)|. 


Wavelets w ;,(¢) or vectors wj, Rescale and shift the time axis to create 
W k(t) = woo(2/t —k). Vectors from woo = (1,1, —1,—1) would be (1,—1,0,0) and 
(0,0,1;=1), 


MATLAB Teaching Codes 


cofactor 
cramer 
deter 
eigen2 
eigshow 
eigval 
eigvec 
elim 
findpiv 
fourbase 
grams 
house 
inverse 
leftnull 
linefit 

Isq 
normal 
nulbasis 
orthcomp 


partic 


Compute the n by n matrix of cofactors. 

Solve the system Ax = b by Cramer’s Rule. 

Matrix determinant computed from the pivots in PA = LU. 
Eigenvalues, eigenvectors, and det(A — A/) for 2 by 2 matrices. 
Graphical demonstration of eigenvalues and singular values. 
Eigenvalues and their multiplicity as roots of det(A — AJ) = 0. 
Compute as many linearly independent eigenvectors as possible. 
Reduction of A to row echelon form R by an invertible E. 

Find a pivot for Gaussian elimination (used by plu). 

Construct bases for all four fundamental subspaces. 
Gram-Schmidt orthogonalization of the columns of A. 

2 by 12 matrix giving corner coordinates of a house. 

Matrix inverse (if it exists) by Gauss-Jordan elimination. 
Compute a basis for the left nullspace. 

Plot the least squares fit to m given points by a line. 
Least-squares solution to Ax = b from ATA = Ab. 
Eigenvalues and orthonormal eigenvectors when ATA = AA!, 
Matrix of special solutions to Ax = 0 (basis for null space). 
Find a basis for the orthogonal complement of a subspace. 


Particular solution of Ax = b, with all free variables zero. 


plot2d 

plu 
poly2sir 
project 
projmat 
randperm 
rowbasis 
samespan 
signperm 
slu 

slv 

splu 

splv 
symmeig 


tridiag 
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Two-dimensional plot for the house figures. 

Rectangular PA = LU factorization with row exchanges. 

Express a polynomial as a string. 

Project a vector b onto the column space of A. 

Construct the projection matrix onto the column space of A. 
Construct a random permutation. 

Compute a basis for the row space from the pivot rows of R. 

Test whether two matrices have the same column space. 
Determinant of the permutation matrix with rows ordered by p. 
LU factorization of a square matrix using no row exchanges. 
Apply slu to solve the system Ax = b allowing no row exchanges. 
Square PA = LU factorization with row exchanges. 

The solution to a square, invertible system Ax = b. 

Compute the eigenvalues and eigenvectors of a symmetric matrix. 


Construct a tridiagonal matrix with constant diagonals a, b, c. 


These Teaching Codes are directly available from the Linear Algebra Home Page: 


http://web.mit.edu/18.06/www. 


They were written in MATLAB , and translated into Maple and Mathematica. 


Linear Algebra in a Nutshell 


(A isn by n) 


Nonsingular 


Singular 


A is invertible. 

The columns are independent. 
The rows are independent. 

The determinant is not zero. 

Ax = 0 has one solution x = 0. 

Ax = b has one solution x = A7'b. 
A has n (nonzero) pivots. 


A has full rank r =n. 


The reduced row echelon form is R =I. 


The column space is all of R”. 

The row space is all of R”. 

All eigenvalues are nonzero. 

A'A is symmetric positive definite. 


A has n (positive) singular values. 


A is not invertible. 

The columns are dependent. 

The rows are dependent. 

The determinant is zero. 

Ax = 0 has infinitely many solutions. 
Ax = b has no solution or infinitely many. 
A has r <n pivots. 

A has rank r <n. 

R has at least one zero row. 

The column space has dimension r < n. 
The row space has dimension r < n. 
Zero is an eigenvalue of A. 

A'A is only semidefinite. 


A has r <n singular values. 


Each line of the singular column can be made quantitative using r. 


Solutions to Selected Exercises 


Problem Set 1.2 


. The lines intersect at (x, y) = (3, 1). Then 3(column 1) + 1(column 2) = (4, 4). 
. These “planes” intersect in a line in four-dimensional space. The fourth plane nor- 


mally intersects that line in a point. An inconsistent equation like u + w = 5 leaves 
no solution (no intersection). 


. The two points on the plane are (1, 0, 0, 0) and (0, 1, 0, 0). 

. Solvable for (3, 5, 8) and (1, 2, 3); not solvable for b = (3, 5, 7) or b = (1, 2, 2). 

. Column 3 = 2(column 2) — column 1.Ifb = (0, 0, 0), then (u, v, w) = (c, —2c, c) 
. Botha = 2 anda = —2 give a line of solutions. All other a give x = 0, y = 0. 

. The row picture has two lines meeting at (4, 2). The column picture has 4(1, 1) + 


2(—2, 1) = 4(column 1) + 2(column 2) = right-hand side (0, 6). 


. The row picture shows four lines. The column picture is in four-dimensional space. 


No solution unless the right-hand side is a combination of the two columns. 


. Ifx, y, z satisfy the first two equations, they also satisfy the third equation. The line 


L of solutions contains v = (1, 1,0), w = G, 1, 5), and u = tu + iw, and all 
combinations cv + dw withc +d = 1. 


. Column 3 = column 1; solutions (x, y, z) = (1, 1, 0) or (0, 1, 1) and you can add 


any multiple of (—1, 0, 1); b = (4, 6, c) needs c = 10 for solvability. 


. The second plane and row 2 of the matrix and all columns of the matrix are changed. 


The solution is not changed. 


23. u = 0,v = 0; w = 1, because 1 (column 3) = b. 
Problem Set 1.3 
1. Multiply by ¢ = 4? = 5, and subtract to find 2x + 3y = 1 and —6y = 6. Pivots 


2, —6. 


. Subtract —} times equation 1 (or add 5 times equation 1). The new second equation 


is 3y = 3. Then y = 1 and x = S. If the right-hand side changes sign, so does the 
solution: (x, y) = (—5, —1). 


. 6x + 4y is 2 times 3x + 2y. There is no solution unless the right-hand side is 


2-10 = 20. Then all points on the line 3x + 2y = 10 are solutions, including (0, 5) 
and (4, —1). 


. Ifa = 2, elimination must fail. The equations have no solution. If a = 0, elimination 


stops for a row exchange. Then 3y = —3 gives y = ~—1 and 4x + 6y = 6 gives 
3. 


. 6x — 4y is 2 times (3x — 2y). Therefore, we need b2 = 2b,. Then there will be 


infinitaly many calutinne The ralamne (2 AY and (9 —A\ ara an tha cama line 


11. 


13. 


15. 


17. 


19. 


21. 


23. 


25. 


27. 


29. 


31. 


2x — 3y =3 2x — 3y =3 x =3 Subtract 2 x row 1 from row 2 


y+ z=I1 gives y+ z=1and y=1 Subtract 1 x row 1 from row 3 
2y — 3z =2 — 5z=0 z=0 Subtract 2 x row 2 from row 3 
The second pivot position will contain —2 — b. If b = —2, we exchange with row 
3. If b = ~1 (singular case), the second equation is —y — z = 0. A solution is 
(1,1, —-1). 


If row 1 = row 2, then row 2 is zero after the first step; exchange the zero row with 
row 3 and there is no third pivot. If column 1 = column 2, there is no second pivot. 


Row 2 becomes 3y — 4z = 5, then row 3 becomes (¢ + 4)z =t —5. If g = —4, 
the system is singular — no third pivot. Then, if t = 5, the third equation is 0 = 0. 
Choosing z = 1, the equation 3y — 4z = 5 gives y = 3 and equation | gives 
x= —9., 

The system is singular if row 3 is a combination of rows | and 2. From the end view, 
the three planes form a triangle. This happens if rows 1 +2 = row 3 on the left-hand 
side but not the right-hand side: for example, x + y+z=0,x-—2y—z = 1, 
2x — y = 9. No two planes are parallel, but still no solution. 

The fifth pivot is g. The nth pivot is ek) 


u+vt+tw= 2 i= 3 
Triangular system 2v + 2w = —2 Solution v = — 
20 2 weiss, af 


(u,v, w) = (3/2, 1/2, —3). Change to +1 would make the system singular (2 equal 
columns). ; 
a = Orequires a row exchange, but the system is nonsingular: a = 2 makes it singular 
(one pivot, infinity of solutions); a = —2 makes it singular (one pivot, no solution). 
The second term be + ad is (a + b)(c + d) — ac — bd (only 1 additional 
multiplication). ; 

Elimination fails for a = 2 (equal columns), a = 4 (equal rows), a = 0 (zero 
column). 


Problem Set 1.4 


1. 


3. 


5. 


7. 


17 5 
4}, |-2), A . With sides to (2, 1) and (0, 3), the parallelogram goes to (2, 4). 
7 3 
3 5 1 
Inner products 54 and 0, column times row gives |-6 —10 —2)}. 
21 35 7 
Ax = (0,0, 0), so x = (2, 1, 1) is a solution; other solutions are cx = (2c, c, c). 
1 0 0 1 3 4 13 4 
Examples: Diagonal |0 2 OJ], symmetric |3 2 OJ, triangular |O 2 Oj}, 
0 0 7 4 0 7 0 0 7 


03 4 
skew-symmetric |—3 0 O|. 
—4 0 0 


27. 


29. 


JL. 


33. 


35. 


37. 
39. 


41. 


43. 


L 
m1 


: Qi 
- (a) ayy (0) fa = 4a/ar (©) new aj; is aj; — at 


il 
: a 
(d) second pivot a. — ate 
i 


. The coefficients of rows of B are 2, 1, 4 from A. The first row of AB is [6 3]. 


0 1 0 1 0 1 1 -l 
ae[2 e=2 JonP omacer=[ —] 


. AB, = BA givesb=c=0. AB, = BoA givesa=d.SoA=al. 
. A(A + B) + B(A+ B), (A+ B)(B + A), A? + AB + BA + B* always equal 


(A +B)’. 


c di|lr s| le d 


AEE ae ble = fess 32 


cp+dr cq+ds 


~_ AP = ASB = f ° } C= F a = Zero matrix. 


0 (-1)" 0 0 


» E32Eyb = (1, —5, —35) but £2;£32b = (1, —5, 0). Then row 3 feels no effect 


from row 1. 


. Changing a3; from 7 to 11 will change the third pivot from 5 to 9. Changing a33 


from 7 to 2 will change the pivot from 5 to no pivot. 
0 


1 0 
To reverse £3), add 7 times row 1 to row 3. The matrix is R3; = {0 1 Oj. 
7 0 


10 1 101 2 POd cl 
Ey,3= {1/0 1 0};;0 1 Of; £3;£i3= {0 1 OJ}. Test on the identity matrix! 
Oy To Oe aA Bp 10) 51 


Ex has 2) = —4, Ex) has 32 = —2, Ey has £4, = —4. Otherwise the E’s match J. 
a+ b+ c= 4 G=2 
a+2b+4c= 8 gives b=1. 
a+3b+9c = 14 eet 
(a) Each column is E times a column of B. 
be Oh id, Qe 2H 12 4 

©) eB=|; i f 2 l= (3 4 ‘| 

Rows of £ B are combinations of rows of B, so they are multiples of [1 2 4], 
(row 3)-x is }> a3;x;, and (A*)1; = (tow 1) - (column 1) = 57 Ay ;j1. 


BA = 31 isSby5, AB = 5/] is3 by 3, ABD = 5Dis3 by 1, ABD: No, A(B+C): 


No. 
00 1 
(a) B=4/7. (b) B=O0. (c) B=j0 1 O}. 
1 0 0 


(d) Every row of B is 1, 0, 0. 
(a) mn (every entry). (b) mnp. (c) n° (this is n* dot products). 


1 0 3°-3° 0 000 20 
2} [3 3 O]+]4/ [12 1]=|6 6 OF +]4 8 4)=]10 14 4 
2 1 6 6 0 i 2.4 qi 


47. 


49. 
51. 


53. 


55, 


57. 


59. 


61. 


viel ||| EJ =I ||| HIE 


The (2, 2) block § = D—CA™'B is the Schur complement: blocks in d — (cb/a). 
A times X = [x, x2 x3] will be the identity matrix J] = [Ax,; Ax 2 Ax3]. 


a+b a+b _ fate b+d]. _ 7 
ae | agrees with cae a when b = c anda = d. 
2x+3y+z2+5t = 8 is Ax = b with the 1 by 4 matrix A = [2 3 1 5]. The 


solutions x fill a 3D “plane” in four dimensions. 


x 
The dot product [1 4 5] }|y{ = (1 by 3)(3 by 1) is Zero for points (x, y, z) ona 
Z 
plane x + 4y + 5z = 0 in three dimensions. The columns of A are one-dimensional 
vectors. 


Axv=[3 4 5] andv’ xv = 50; vA gives an error message. 


8 3 4 5-+u 5—ut+yv 5-—v 
M=!/1 5 9| = {5—-u-—v 5 Stutvl; 
6 7 2 5+ Stu-v 5-u 


Ms(1, 1, 1) = (15, 15, 15); Ma(1, 1, 1, 1) = (84, 34, 34, 34) because the numbers 
1 to 16 add to 136, which is 4(34). 


Problem Set 1.5 


1. 


U is nonsingular when no entry on the main diagonal is zero. 


1 OO 0 1 0 OO; j1 0 O 1 0 0 1 0 0 
2 1 O};]-2. 1 OJ=jO 1 OF;]/-2 1 0 2 1 O}=Talso. 
—-l -1 ij}}-1 1 1 0 0 1| {-1 1 1}]-1 -1 1 


(E~ F-1G-))(G FE) = EF" FE = EE = J; also(GFE)(E? FG") =I. 


1 0-0); ]2> 3° 33 23 3) a 2 
~LU={0 1 O} |0 5S 7); afterelimination, }0 5 7] jv] = : 
3 0 1; |0 O —-1 0 0 —1l| jw —1 
10 0 0 10 0 0 
Bi Or 2 1 0 0 
. FGH= 021 op HGF = 421 0 
9 ie ee a 8 4 2 1 
- (a) Nonsingular when d,d.d; 4 0. (b) Suppose d; # 0: Solve Lc = b going 
0 d, —d, 0 Uu 0 
downward: Le = b givesc = |0|.Then |O0 dz, -—d)} |v] = {0} gives 
1 © Oe ds w 1 
1/d; 
x= 1/d; . 


1/ds 


11. 


13. 


15. 


17. 


19, 


21. 


23. 


25. 


27. 


2 2 


Le = b going downward gives c = | —2|; Ux = c upward gives x = |—2}]. 
0 0 
, 1 0 0 Uu 2 
Permutation 
> and 3 0 0 lLivi= /-3|; 
rows 2 an 01.0 +i A 
facie 0 1 (0 1 
panera 1.0 Oh lela f= 
rows | an 00 0 i 1 
Oo 1 0} JO 1 1 1 0}; 11 O O|} j1 O 1 
PA=LDUisij1 0O O} j1 O 1; =j0 1 O;| JO 1 O; }0 1 If; 
0 0 1) /2 3 4 2 3 1t/0 0 -1;} 10 O 1 
1 0 0); J}1 2 1 1 0 Of j1 0 O}; j1 2 1 
PA=LDUis|0 0 1 4 =f{t1 1 0/430 -—1 Of; JO 1 Ot. 
0 1 0 1 O 1; JO 0 0} }0 O O 
1 0 0 
L becomes |1 1 0}. MATLAB and other codes use PA = LU. 
2 «0 


a = 4 leads to a row exchange; 3b + 10a = 40 leads to a singular matrix; c = 0 
leads to a row exchange; c = 3 leads to a singular matrix. 

£3; = 1 and £32 = 2 (£33 = 1): reverse steps to recover x + 3y + 6z = 11 from 
Ux =c: 1 times (x + y+z=5) +2 times (y + 2z = 2) + 1 times (z = 2) gives 
x-+3y +6z = 11. 


J = 0 is not allowed 


2by 2:d 

1 1 0 if deg a=1,. e¢= 1, then’ =1 
a fh 

2 i no pivot in row 2. 


2 4 8 2 
A=10 3 9} hasL=JandD= 3 ;A = LU has U = A (pivots on 
0 7 


Ly 2 
the diagonal); A= LDU hasU = D™'A= |0 1 3) with 1s on the diagonal. 
0 0 


29. 


31. 


33. 


35. 


37. 


39. 


41. 


43. 


45. 


47. 


aaaa 1 a a a a a #0 
ab b b 1 1 b-—-a b-a b-a ba 
Ge eR ea Be CNS ey 
abed 1111 d-c d#c. 
1 a a a 0 a 
| 1 1] =ZIU; ja a+b b = (same L) b 
01 1 1 0 jb bte c 
(same U). 
1 0 0 4 4; /1 1 1 4 3 
1 1 O|c= {5] givesc= |1].;/0 1 1} x= {1 givesx = [0]. 
ae ee 6 1} jO0 O 1 i 1 
A= LU. 
The 2 by 2 upper submatrix B has the first two pivots 2, 7. Reason: Elimination on 


A starts in the upper left corner with elimination on B. 


Lk vd dD. al 
12 3 4 5 
LS oo 10-15 
1 4 10 20 35 
L. S. d5 35. 70 
1 | Sao GR ae (ea | Pascal’s triangle in L and U. 
tid 12 3 4 MATLAB’s lu code will wreck. 
71) Qed 1 3> 6]. the pattern. chol does no row 
£03 I 1 4 exchanges for symmetric 
146 4 1 1 matrices with positive pivots. 
Each new right-hand side costs only n? steps compared to n?/3 for full elimination 


A\b. 
2 exchanges; 3 exchanges; 50 exchanges and then 51. 


010 100 001 
P=1/]0 0 1/;P; = {0 O 1j andP,=|0 1 O 
10 0 010 i 8:6 


(P gives a column exchange). 


There are n! permutation matrices of order n. Eventually two powers of P must be 
the same: If P’ = P* then P’~* = I. Certainlyr —s <n! 


| 0 1 0 
-Pj={0 0 1|,and Po = 7. 
1 0 100 


Par ; ey ae 
p=| 5 is Sby 5 with y= | 


The solution is x = (1,1,..., 1). Then x = Px. 


Problem Set 1.6 


I. 


jon) 


jas 7 | cos@ el 


-1 0 3 Az} 5 
aie 5 or <2 =| f —sin@ cosé 


3. Av} = BC; A“! = U“!L"'P. 


11. 


13. 
15. 


i; 


19. 


21. 


23. 
25. 


27. 
29. 


31. 


. A(AB) = (move parentheses) = (A*)(B) = I. 


ee a me | a 


0 1 
12. 2/372 1/2 Apt | | at nave a? = 7 


. If row 3 of A~! were (a, b,c, d), then A7'A = J would give 2a = 0,a + 3b = 0, 


4a + 8b = 1. This has no solution. 
1 0 —1 0 0 0 1 0 0 0 1 0 
(a) F i+ | ; a, | (b) a 01+ [0 = (9 ih 
me E tL f= [1 | READE BA BA 
2. 2, 6.3 


(a) n(n + 1)/2 entries on and above diagonal. (b) (7 — 1)n/2 entries above 
diagonal. 


A'B =8:B'A=8:AB' = ; i BA = I 


(a) The inverse of a lower (upper) triangular matrix is still lower (upper) triangu- 
lar. Multiplying lower (upper) triangular matrices gives a lower (upper) triangular 
matrix. (b) The main diagonals of Laks Dy and D,U, Uy ' are the same as those 
of D2 and D, respectively. L7'L2D, = D,U,U;', so we have D,; = D2. By com- 


paring the off-diagonals of L7'L,D, = D,U,U,;', both matrices must be diagonal. 


L7'LyDz = Dy, DyxU;Uz' = Dy, Dy is invertible so L7'Ly = I, UjUy) = 1. 
Then Ly = Lo, U; = U>. 

TOOT: OF OF SS 

3 1 Of] jo 3 oO} jo 1 1); 

SPE ONO Dy 0: Oe at 


nb 8), Fe 0 1 b/a] _ 
a i F as E ("| = Et 


From BUI — AB) = (I — BA)B we get (J — BA)"! = BUI — AB)7'B"!, an 
explicit inverse provided B and J — AB are invertible. Second approach: If I— BA 
is not invertible, then BAx = x for some nonzero x. Therefore ABAx = Ax, or 
ABy = y, and J — AB could not be invertible. (Note that y = Ax is nonzero, from 
BAx =x.) . 


x a a —.2 ot 5 -2 

E-Lab Y-[ales'=a le 7} 
(a) In Ax = (1,0, 0), equation 1 + equation 2 — equation3isO = 1. (b) The 
right-hand sides must satisfy b} +b, = b3.  (c) Row3 becomes arow of zeros—no 
third pivot. 
If B exchanges rows 1 and 2 of A, then B~' exchanges columns 1 and 2 of A~!. 


If A has a column of zeros, so does BA. So BA = IJ is impossible. There is no A7! 
1 1 1 


1 1 —1 1 = i-l 1 =F > 
—1 1} j-l 1 1 0 —-i 1 
1 
then |}1 1 is L = E~', after reversing the order of these 3 elementary matrice 
ee ee 


and changing —1 to +1. 


oa: 


35. 


37. 


39. 


41. 


43. 


45. 


47. 


49, 


51. 


53. 


55. 


a7. 
59, 


A * ones(4,1) gives the zero vector, so A cannot be invertible. 


io 5 0s i Oe a es 
ee ak i) (oe i] =O A“ 


ee: 0-8 3 
oeiaae 1 ea a 
Los OO. Pa a Oe 
OE GeOrd OS) AGO ae 
O70 2.0 ih AO, Ose 6 Oe. 4 
1 00 1 -a ac~b 
2101-0. 05 Bel. 
0010 0 


De ON OO e0, “A EO ON nee: as 
eee ae leon 1/2 |= aH 


Not invertible for c = 7 (equal columns), c = 2 (equal rows), c = 0 (zero column). 
1 1 0° 0 . 
=| 0 1 1 0 ay ? . 
A= 001 1f The 5 by 5 A~ also has Is on the diagonal and super- 
00 0 1 
diagonal. 


I O Au! ‘0 aa —-D I 
SO FSD Cast poh. 1 OF 
For Ax = b with A= ones(4, 4) = singular matrix and b = ones(4, 1), A\b 


will pick x = (1,0,0,0) and pinv(A) * b will pick the shortest solution x = 
(1, 1, 1, 1)/4. 


a= |) - Atli= 3 ma (AHF = aS), Ak A™ = Aand 


1 {0 
=p IT = (AT)-1. 
then A = ° i|- (A~*)* = (A’*) 
((AB)"!)" = (B-1A7!)? = (A)T(B-!)'; (U7!) is lower triangular. 
G3 T p 
(a) xTAyY =ay =5. (b) xTA=[4 5 6]. () Ay= B 
(Px)'(Py) = x™P™Py = x'y because P™P = J; usually Px-y =x-Ply # 
x- Py: 
0 1 0} f1 1 ] 0 1 Of; [1 
0 0 Lp 12) 41) se 12-40 0 2 1 
1 0 Of 43 2 3 1 0 0 
PAP" recovers the symmetry. 


(a) The transpose of R™AR is R'ATR™! = R™AR =n byn. 
(b) (R™R);; = (column j of R) - (column j of R) = length squared of column j. 


61. 


63. 
65. 


67. 


69. 
71. 


1 O 1] {ye yac + yes 
Total currents are Aly = j-l 1 O} | ycs| = |~ysc + Yes}. 
O —1 ~—1] | yes —Ycs — Yes 


Either way (Ax)! y = x" (ATy) = xpysct+xpyss— Xcysc t+ xcYcs — Xs Yes — Xs Yas. 
Ax - y is the cost of inputs, whereas x - A’y is the value of outputs. 
These are groups: Lower triangular with diagonal 1s, diagonal invertible D, and 
permutations P. Two more: Even permutations, all nonsingular matrices. 
i 
‘| 


Reordering the rows and/or columns of E A will move entry a, not giving E 


Random matrices are almost surely invertible. 
The —1, 2, —1 matrix in Section 1.7 has A = LDL! with €,;-; = 1 — +. 


Problem Set 1.7 


2 —1 
—] 2 —1| 
I. —] 2 -1 
—] 2 
1 2 1 —} 
ae 1 a 1 —2 - 
4 det = 5 
3 5 : 
td ri 1 
1 -1 (S 0 
—1 2 —1 C 0 
3. Ap = —1 2 —-1 . Each row adds to 1,so Ap |c}| = |O]. 
—] 2 —1 c 0 
—1 1 c 0 
5. (uy, Up, U3) = (17/8, 0, ~27/8) instead of the true values (1, 0, —1). 
9 —36 30 
7. H7'=|-36 192 —180]. 
30 —180 180 
9. The 10 by 10 Hilbert matrix is very ill-conditioned. 
11. A large pivot is multiplied by less than 1 in eliminating each entry below it. An 
1/2 1/2 1 
extreme case, with multipliers = 1 and pivots = 4,5,4,isA= ]-1/2 0 1). 
—1/2 -1 1 
Problem Set 2.1 
1. (a) The set of all (u, v), where u and v are ratios p /4 of integers. (b) The set of 


3. 


all (u, v), where u = Oorv = 0. 


C(A) is the x-axis; N(A) is the line through (1, 1); C(B) is R?; N(B) is the line 
through (—2, 1, 0); C(C) is the point (0, 0) in R’; the nullspace-W (C) is R?. 


. Broken rules: (a) 7, 8 (b) 1 (c) 1, 2, 8. 


7. (b), (d), (e) are subspaces. Can’t multiply by —1 in (a) and (c). Can’t add in (f). 
9. The sum of two nonsingular matrices may be singular (A + (—A)). The sum of two 
singular matrices may be nonsingular. 

11. (a) One possibility: The matrices cA form a subspace not containing B. 

(b) Yes: the subspace must contain A — B = I. 
(c) The subspace of matrices whose main diagonal is all zero. 

13. If (f + g)(x) is the usual f(g(x)), then (g + f)x is g(f(x)), which is different. In 
rule 2 both sides are f (g(h(x))). Rule 4 is broken because there might be no inverse 
function f~!(x) such that f(f~!(x)) = x. If the inverse function exists, it will be 
the vector —f. 

15. The sum of (4, 0, 0) and (0, 4, 0) is not on the plane; it has x + y — 2z = 8. 

17. (a) The subspaces of R* are R? itself, lines through (0, 0), and the point (0, 0). 
(b) The subspaces of R‘ are R’* itself, three-dimensional planes n-v = 0, two- 
dimensional subspaces (n; - v = 0 and 2 -v = 0), one-dimensional lines through 
(0, 0, 0, 0), and (0, 0, 0, 0) alone. 

19. The smallest subspace containing P and L is either P or R?. 

21. The column space of A is the x-axis = all vectors (x, 0, 0). The column space of B 
is the x-y plane = all vectors (x, y, 0). The column space of C is the line of vectors 
(e220): 

23. A combination of the columns of C is also a combination of the columns of A (same 
column space; B has a different column space). 

25. The extra column b enlarges the column space, unless b is already in that space: 
(Abs 1 01 (larger column spate) 

~ 10 0 1 (no solution to Ax = D). 
1 0 1 (b already in column space) 
01 1 (Ax = b has a solution). 
27. Column space = R®. Every b is a combination of the columns, since Ax = b is 
solvable. 
1 1 0 by: le 32 12 0 
29. A=il O Of} or/1 O 1/;A= {2 4 OF (columns on 1 line). 
0 1 0 01 1 3 6 0 
31. R’ contains vectors with two components—they don’t belong to R°. 
Problem Set 2.2 
lL x+y+z=1,x+y+z=0. Changing 1 to0, (x, y, z) =c(—1, 1,0) + d(—1, 0, 1). 
0 1 0 3 . ; : 
3. Echelon form U = 0 0 0 of free variables x;,x3,x4; special solutions 


(1, 0, 0,0), (0,0, 1,0), and (0, —3, 0, 1). Consistent when b, = 2b;. Complete 
solution (0, b;, 0,0) plus any combination of special solutions. 


wv 


11. 


13. 


15. 
17, 


19, 


21. 


23. 
25. 


27. 


29. 


Uu —2v —3 —2 —3 
vil = v =v 1} + | 0}; no solution! 
w 2 0 2 


. c= 7 allows u = 1,v = 1, w = 0. The column space is a plane. 


a : 120 -2 
5. (a). KESS XS + x4 , for any X2, x4. Row-reduced R=|0 O 1 ot, 
0 —2 
0 0 0 0 
0 1 
a— 3b —2 2 
P 0 1 0 
(b) Complete solution x = i + x2 0 + X4 _9) for any x2, x4. 
0 0 1 


f 1 Py - a has nullspace = line through (—1, 1) but no solution. Any 


= < has many particular solutions to Ax, = b. 


‘re ee ee | Ae es 1-1 1 -1 (a) r=1. 
R=|0 0 0 O;R=1|) 9 9 of R=/0 00 0O (b) r= 2. 
0 0 0 0 0000 0 00 Of ©r=1. 
a ae 
A nullspace matrix N -| A isn byn—r. 
I think this is true. 
se 1 0 
The special solutions are the columns of N = 1 0 and N=|0 —2}. 
Oo 1 
oo 4 


The r pivot columns of A form an m by r submatrix of rank r, so that matrix A* 
has r independent pivot rows, giving anr by r invertible submatrix of A. (The pivot 
rows of A* and A are the same, since elimination is done in the same order—we 
just don’t see for A* the “free” columns of zeros that appear for A.) 


T 


(uv')(wz!) = u(v'w)z' has rank 1 unless v'w = 0. 


We are given AB = I which has rank n. Then rank(AB) < rank(A) forces 
rank(A) =n. 


If R = EAand the same R = E*B, then B = (E*)~'EA. (To get B, reduce A to R 
and then invert steps back to B.) B is an invertible matrix times A, when they share 
the same R. 


Since R starts with r independent rows, R? starts with r independent columns (and 


then zeros). So its reduced echelon form is h ] where J isr by r. 


31. 


33. 


35. 


37. 
39. 


41. 


43. 


45. 


47. 


1. 1 2 2 
Ife=1,R= 10 0 O O| has x, x3, x4 free. 
00 0 0 
t 072.2 
Ifc#1,R= 10 1 O O}| has xs, x, free. 
00 0 0 
-l1 —2 -2 —2 —2 
se teegtnt oe ae T Ste THO. SO ee ee! 
Special solutions in N = 0 1 0 (c =1)andN = 1 0 (c 4 1) 
0 Oo 1 0 1 


Tos Li Rs i 


c #1,2. 
Special solutions in N = a (c= l)oN= H (c = 2) or N = 2 by 0 empty 


1 has x, free; ifc = 2, R= f Be 


00 0 | has x, free; R = J if 


matrix. 


2) [3 We hecctl illecells 
Xcomplete = O} + x2 1 >Xcomplete = 1/2 + X2 0 + X4 9} 
: : 0 0 1 
(a) Solvable if b. = 2b, and 3b, — 3b; +b, = 0. Then x = ies za 
3 — 2D) 
(no free variables). (b) Solvable if b = 2b; and 3b, — 3b3 + bg = 0. 
5b, — 2b; —1 
Then x = | b3 — 2b, | + x3 |-1}. 
0 1 


A 1 by 3 system has at least two free variables. 
(a) The particular solution x, is always multiplied by 1. (b) Any solution can 


bex,. (Cc) F 4 | oa [| Then i is shorter (length /2) than Al (d) The 


“homogeneous” solution in the nullspace is x, = 0 when A is invertible. 
Multiply x, by 2, same x,; EB is i special solutions also include the columns 
P 


of gE. x» and the special solutions are not changed. 


For A, g = 3 gives rank 1, every other g gives rank 2. For B, g = 6 gives rank 1, 
every other g gives rank 2. 


(a) r<m,alwaysr <n. (b) r=m,r<n. (Cc) r<m,r=n. (d) r=m=n. 


100 0 0 100 -1 
R= |0 0 1 Oj andx, = [1]; |0 O 1 2): no solution because of 
00 0 0 0 00 0 5 


row 3. 


49, 


51. 


53. 


5S, 


Bi: 


59. 


61. 


63. 


65. 


67. 


69. 


1 1 

A= |0 2]; B can’t exist since two equations in three unknowns can’t have one 
0 3 

solution. 

A has rank 4 — 1 = 3; the complete solution to Ax = Ois x = (2,3, 1,0). 


1 0 -2 0 
R={|0 1 -3 O| with —2, —3 in the free column. 
0 O 0 1 


(a) False. (b) True. (c) True (onlyn columns). (d) True (only m rows). 


Oe at ee At Oe A Th oO" et 
UV = 0001111 god es 0 0 1 0 1 1] (R doesn’t come 
+O OOO Ld 4 ~ 10 000 1 1 1] from this U). 
0000000 000000 0 
If column 1 = column 5, then x5 is a free variable. Its special solution is 


(—1, 0, 0, 0, 1). 

Column 5 is sure to have no pivot since it is a combination of earlier columns, and x5 
is free. With four pivots in the other columns, the special solution is (1, 0, 1, 0, 1). 
The nullspace contains all multiples of (1, 0, 1, 0, 1) (a line in R>). 


100 -—4 
A 0) 1. O34; 
0°01 2 


This construction is impossible: two pivot columns, two free variables, only three 
columns. 


0 1 
ao k of 
R is most likely to be J; R is most likely to be J with fourth row of zeros. 


Any zero rows come after these rows: R = [1 —2 —3], R = E ) 4 


R=I. 


Problem Set 2.3 


1. 


3. 


1 1 tf Je 
O 1 1| Jeo] = 0 gives c3 = co = cy = O. But yy + vp — 403 + vy = 0 
0 0 TY je 
(dependent). 
If a = O then column 1 = 0; if d = 0 then b(column 1) — a(column 2) = 0; if 
f = 0 then all columns end in zero (all are perpendicular to (0, 0, 1), all in the xy 
plane, must be dependent). 


19. 


21. 
23. 


25. 


27. 


29. 
31. 


33. 


oO 
oS 


3 
car 
=5 


invertible = independent columns 
(could have used rows). 


. (a) Line in R’. 
. All dimensions are 2. The row spaces of A and U are the same. 


~18/5 


1 2 -3 LZ 
=3 1 21|—> |0 7 
2 -—3 1 


—3 1 
mat gee. Ge eee 
0 0 0 i 


columns add to 0 
(could use rows). 


0 
(b) Of, 
0 


. The sum Vy — V2 + v3 = O because (w, — w3) — (Ww; — w3) + (wW; — W2) = 0. 


. (a) The four vectors are the columns of a 3 by 4 matrix A with at least one free 


variable, so Ax = 0. (b) Dependent if [v, v.] has rank 0 or 1. 
(c) Ov; + c(O, 0, 0) = 0 has a nonzero solution (take any c ¥ 0). 


(b) Plane in R*®. (c) Planein R®. (d) Allof R’. 


.v=5(v+w) +$(v—w) and w = 4(v + w) — $(u — w). The two pairs span the 


same space. They are a basis when v and w are independent. 


. If elimination produces one or more zero rows, the rows of A are linearly dependent; 


1 0 0 
—1 
0 
0 


for example in Problem 16 — 


1 

1 0 1 0 
Oe) ee | 
0101 


—1 
0 
0 

The n independent vectors span a space of dimension n. They are a basis for that 

space. If they are the columns of A then m is not less than n (m > n). 


C(U): Any bases for R?; N(U): (row 1 and row 2) or (row 1 and row 1 + row 2). 


Independent columns => rank n. Columns span R” => rankm. Columns are basis 
for R” => rank =m =n. 


Ooo e © coo = 
— 

Corer CO — pee 

orf OO Fe oO 


(a) The only solution is x = 0 because the columns are independent. (b) Ax = b 
is solvable because the columns span R?. 

Columns 1 and 2 are bases for the (different) column spaces of A and U; rows 1 and 
2 are bases for the (equal) row spaces; (1, —1, 1) is a basis for the (equal) nullspaces. 
rank(A) = 2 if c = 0andd = 2; rank(B) = 2 except when c = d orc = —d. 

Let v; = (1,0,0,0),..., vg = (0,0, 0, 1) be the coordinate vectors. If W is the 
line through (1, 2, 3, 4), none of the v’s are in W. 

(a) If it were not a basis, we could add more independent vectors, which would 
exceed the given dimension k. (b) If it were not a basis, we could delete some 
vectors, leaving less than the given dimension k. 


35. 


37. 


39. 
Al. 


43. 


45. 


(a) False, might be no solution. (b) True, 7 vectors in R° are dependent. 


1 0 0 0 0 0 0 0 0 Oo 1 0} {0 0 1 
(a) 10 O O}, {0 1 Of, |O O O}. (b) Add j1 O OF}, |0 O 0}, 
0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 
0 0 0 0 1 0 00 1 0 oO 0 
0 0 lf. (c) |-1 0 O}, | O O OF}, |O O 1) are a basis for all 
0 1 0 0 0 OO} |-1 0 0 0 -1 0 
A=-—Al. 
y(0) = Orequires A+ B+C = 0. One basis is cos x — cos 2x and cos x — cos 3x. 
yi (x), yo(x), y3(x) canbe x, 2x, 3x (dim 1) or x, 2x, x* (dim 2) or x, x”, x3 (dim 3). 
1 1 1 1 1 
1 =|] — 1| + 1 + 1 
1 1 1 1 1 
1 
—|1 
1 


Check the (1, 1) entry, then (3, 2), then (3, 3), then (1, 2) to show that those five P’s 
are independent. Four conditions on the nine entries make all row sums and column 
sums equal: row sum | = row sum 2 = row sum 3 = column sum 1 = column sum 
2 (= column sum 3 is automatic because sum of all rows = sum of all columns). 


If the 5 by 5 matrix [A ] is invertible, b is not a combination of the columns of A. 
If [A b] is singular, and A has independent columns, b is a combination of those 
columns. 


Problem Set 2.4 


. False, we only know dimensions are equal. Left nullspace has smaller dim = m —r. 
«CA r= 2, d,0, 1.0, 1,0); NAb a=—r= 2, 2,—1,1, 0),:C1,0;,0, 1); 


C(A!): r = 2, (1, 2,0, 1), (0, 1,1,0); N(A™):m~r =1, (-1,0, 1); 
C(U): (1, 0,0), (0, 1,0); N(U): (2, -1, 1,0), (-1, 0, 0, 0); 
C(U"): (1,2, 0, 1), (0,1, 1,0); N(A™): @,0, 1). 


. A times every column of B is zero, so C(B) is contained in the nullspace N(A). 


7. From Ax = 0, the row space and the nullspace must be orthogonal. See Chapter 3. 


12 4 


. {1 2 4); 12 4 8] has the same nullspace. 


a 30, 12 


. If Ax = Ohas a nonzero solution, then r < n and C(A?) is smaller than R”. So 


A'y = f is not solvable for some f. Example: A= [1 1] and f = (1, 2). 
. ad = bc/a; the only pivot is a. 
. With independent columns: rank n; nullspace = {0}; row space is R”; /eft inverse. 
-A=[1 1 0O]);B=[0 O 1]. 


19. 


21. 


23. 


25. 


27. 


29. 


31. 
33. 


35. 


37. 


39. 


41. 


No—for example, all invertible n by n matrices have the same four subspaces. 


1 0 
(a) {1 O|. (b) Impossible: dimensions 1+1+43. (c) [1 1}. 
0 1 


| 
Then m—r=n—r. 


(d) Ee “i (e) Impossible: Row space = column space requires m = n. 


Invertible A: row space basis = column space basis = (1, 0, 0), (0, 1, 0), (0, 0, 1); 
nullspace and left nullspace bases are empty. B: row space basis (1, 0, 0, 1, 0, 0), 
(0,4, 0,0, 1, 0), and (0, 0, 1, 0,0, 1); column space basis (1, 0, 0), (0, 1, 0), (0, 0, 1); 
nullspace basis (—1, 0, 0,1, 0,0), (0, —1,0,0,1,0), and (0,0, —1,0,0, 1); left 
nullspace basis is empty. 

(a) Same row space and nullspace. Therefore rank (dimension of row space) is 
the same. (b) Same column space and left nullspace. Same rank (dimension of 
column space). 

(a) No solution means that r < m. Alwaysr <n. Can’t compare m and n. 

(b) If m —r > 0, the nullspace of A‘ contains a nonzero vector. 

Row space basis (1, 2, 3, 4), (0, 1, 2, 3), (0, 0, 1, 2); nullspace basis (0, 1, —2, 1); 
column space basis (1, 0, 0), (0, 1, 0), (0, 0, 1); left nullspace has empty basis. 

If Av = 0 and v is a row of A then v - v = 0. Only v = 0 is in both spaces. 

Row 3 — 2 (tow 2) + row 1 = zero row, so the vectors c(1, —2, 1) are in the left 
nullspace. The same vectors happen to be in the nullspace. 

(a) wand w span C(A). (b) vandzspanC(A!). (c) rank < 2 if u and w are 
dependent or v and z are dependent. (d) The rank of uv! + wz" is 2. 

(a) True (same rank). (b) False (A = [1 OJ). (c) False (A can be invertible 
and also unsymmetric). (d) True. 

ay = lay = 0,413 = 1, ay) = 0,43. = 1,43; = 0, a3 = 1, a33 = 0, a2) = 1 
(not unique). 

Rank r = n means nullspace = zero vector and x, = 0. 


Problem Set 2.5 
1 -1 0 1 
TA =. 10 1-1; N(A) contains multiples of |1|; N(A‘) contains multiples 
1 0 -1 1 
1 
of | 1]. 
—1 
3. The entries in each row add to zero. Therefore, any combination will have that 


same property: fitft+h=GiA y=f>y+y=f,-ntyw = fr, 
—y. ~y3 = f3 => fi + fot fs = 0. It means that the total current entering from 
outside is zero. 


Cy +03 —Cj 
5. —Ci Cy +c2 =aC9 : 


Cytc3 Cy —C3 
: i ered has pivots c; + ¢3, 
ue os 1 3 
—C3 saa 7 a ©) Phe | 
C103 + C1C2 + C2C3 
Cj + C3 


7. Conditions on b are bj + by — bs = 0, bs — bg + bg = 0, Do — bs +g =O. 


3 -1] -1 -!1l Ci + C2 + C5 —C{ maa S29) —C5 
9 —] 3 ~-] ~—1 —C] Cr tco3 +04 —C3 —C4 
“ }-1 ~1 3-1) —C2 —C3 C2 +03 + C6 —C6 
—-1 -1 -1l 3 —Cs5 —C4 —C6 C4 + C5 + 6 
Those c’s that connect to node j will appear in row j. 
1 0 0 0 —1 if Ol iy 0 
0 ¢ 0 O -1 0 1) Ty, 0 a ~3 
0 0 $ 0 0 1 Of} fy 0 = 4 
11. 0 0 0 1 0 0 ~1 V6 | St Oa ah 7 
—1 —1] 0 0 0 0 O} |x fi 3 3 
1 OO 1 0 0 0 Of Ix fo 0 a 
0 1 0 -!I 0 QO O} jx fs 


13. There are 20 choices of 3 edges out of 6 because “6 choose 3” = oa = 20. Four 
choices give triangles, leaving 16 spanning trees. at 

15, I think it is already built in. 

17. 9 nodes — 12 edges + 4 loops = 1; 7 nodes — 12 edges + 6 loops = 1. 

19. x = (1, 1,1, 1) gives Ax = 0; then ATAx = 0; rank is again n — 1. 


and 


1 
u.u=ali ° 

1 
1 


1 
1 
1 
0 


—_=_= Oe 


2 (M?);; = ajayj + +++ + AinQnj 
2 and we get a;,a;; = 1 when there 
2} isa2-step path7 tok to j. 

a Notice 3 paths from a node to itself. 


NN W dN 
NWN Y 


Problem Set 2.6 


; 0 —1| {0 O 
1. Rotation f a k | 


3. ||Ax||? = 1 always produces an ellipse. 


5. They are transformed to (1, 3), (2, 6), (—1, —3). The x-axis turns; vertical lin 
shift up/down but stay vertical. 


17. 


19, 
21. 


23. 
25. 
27. 
29. 
31. 


33. 


32. 


37. 


39. 


41. 


00 2 0 

Second 000 6 
F derivative 0000 The nullspace is spanned by (1,0,0,0) and 

matrix 000 0 


(0, 1, 0, 0), which gives linear P,. Second derivatives of linear functions are zero. 
The column space is accidentally the same as the nullspace, because second deriva- 
tives of cubics are linear. 


. e’ ande™ are a basis for the solutions of u” = u. 


fae Taal ae Bear a k ‘ ee eas 


sinO cos@||sin@d  cosé 0 1 


» (a) Yes. (b) Yes. We don’t need parentheses (AB)C or A(BC) for ABC! 


1 
0 : : and A* = J; the double transpose of a matrix gives 
A= the matrix itself. 
0 1 0 0 of a 
0001 Note Ax3 = 1 because transpose of matrix 2 is matrix 3. 
eo rom Poet nos 
A = ;B=/0 0 1 O|;;,AB= -BA=|0 1 OJ}. 
0 1 0 0001 00 1 0 001 
00 1 ey 000 1 
(a) is invertible with T-'(y) = y'/?; (c) is invertible with T~'(y) = y — 11. 


With w = 0, linearity gives T(v +0) = T(v) + T (0). Thus 7 (0) = 0. Withe = —1, 
linearity gives T(—0) = —7 (0). Certainly T(—0) = T (0). Thus 7 (0) = 0. 
S(T(v)) = S(v) = v. 
(b) and (c) are linear, (a) fails T(2v) = 27 (v), (d) fails T(v + w) = T(v) + T(w). 
T(T(v)) = (vz, 1, v2); T?(v) =v; T(v) = T(T?(v)) = T(v). 
(a) T(1,0) =0. (b) (0,0,1) isnot inthe range. (c) T(0,1) =0. 
Associative law gives A(M,; + Mz) = AM; + AM). Distributive law over c’s gives 
A(cM) = c(AM). ; 

: 0 0 0 1 
No matrix A gives A 1 ol = Io . To professors: The matrix space has 
dimension 4. Linear transformations on that space must come from 4 by 4 ma- 
trices (16 parameters). Those multiplications by A in Problems 31 and 32 were 
special transformations with only 4 parameters. 
0 b 


TU) =O0butM= f 0 


| = T(M); these fill the range. M = é dl in the kernel. 


sf 
(a) M = i ‘| (b) N= e i . (c) ad =be. 


Reorder basis by permutation matrix; change lengths by positive diagonal matrix. 
1 a a*|[A 4 
1 bb?! |B} = |5|; Vandermonde determinant = (b~—a)(c—a)(c —b); the 
lc clic 6 

points a, b, c must be different, and then determinant + 0 (interpolation is possible). 


43. 


45. 
47. 


49. 


If T is not invertible, then T(v,), ..., J (vy) will not be a basis. Then we couldn’t 
choose w; = T(v;) as output basis. 

S(T (v)) = (-1, 2) but S(v) = (—2, 1) and T(S(v)) = (1, —2). SoTS ¥ ST. 
The Hadamard matrix H has orthogonal columns of length 2. So the inverse of H 
is H'/4= H/4. 

False: the m nonzero vectors would have to be independent. 


Problem Set 3.1 
1. |x|] = 21; lly] = 32; xTy = 0. 


23. 


25. 


27. 


29. 


31. 


33. 


. (1, 2, —1) is perpendicular to P. A = | 


» (x2/x,)(¥2/y1) = —1 means that x;y; + x2y2 = 0, soxty = 0. 

. v; and v3 arte orthogonal, also v2 and v3. 

» x=(-2, 1,0); y=(-1, -1, 1); the row z = (1, 2, 1) is orthogonal to the nullspace. 
. The orthogonal complement is the line through (—1, —1, 1) and (0, 0, 0). 

. If ATy = 0, then yb = y'Ax = (y'A)x = 0, which contradicts yb + 0. 

. The figure splits any y in R” into column space part + left nullspace part. 

. No such matrix, because (1, 2, 1)"™(1, —2, 1) £0. 

. The matrix with the basis for V as its rows. Then the nullspace is Vt = W. 

. (a) If V and W are lines in R*, V' and W° are intersecting planes. (b) V. 


1 1 3 


01 3) nas NA) =P: B = [1 2 —-1] 


has row space = P. 
A= E i has subspaces = four lines; (1, 1) orthogonal to (—1, 1), (1, 2) 


orthogonal to (—2, 1). Always row space L nullspace. 


1 2 -3 2 1 1 

(a) 2 -—3 1}. (b) {—3] is not orthogonal to }1{. (c) [1] in C(A) 
—3 5 —2 5 1 1 
: 1 -1l 

and |O} in N(A‘) is impossible: not perpendicular. (d) A = ; S| has 
0 


A*=0. (e) (1, 1, 1) will be in the nullspace and row space; no such matrix. 

(a) If Ax = b hasa solution and Aly = 0, then b'y = (Ax)'y =0 

(b) b is not in the column space; so, not perpendicular to all y in the left nullspace. 
x =X, +Xn, where x; is in the row space and x, is in the nullspace. Then Ax, = 0 
and Ax = Ax, + Ax, = Ax,. All vectors Ax are combinations of the columns 
of A. If x = (1, 0), then x, = (1/2, 1/2). 

(a) For a symmetric matrix, the column space and row space are the same. 

(b) x is in the nullspace and z is in the column space = row space: so these 
“eigenvectors” have x'z = 0. 


x splits into x, +x, = (,-D +d, 1) = @, 0). 


35. Ax = Bx means that[A B] Ei = 0. Three homogeneous equations in four 


unknowns always have a nonzero solution. Here x = (3,1) and x = (1,0), and 
Ax = Bx = (5,6, 5) is in both column spaces. Two planes in R? (through zero) 
must intersect in a line at least! 


37. Aly = 0 gives (Ax)'y = x'ATy =0. Then y L Ax and N(A‘) 1 C(A). 


39. S* is the nullspace of A = ; | Therefore S+ is a subspace even if S is not. 


41. If Vis all of R*, then V~ contains only the zero vector. Then (V+) = R* = V. 
43. (1,1, 1, 1) is a basis for PLA= [i 1 1 1] has the plane P as its nullspace. 
45. Column 1 of A~ is orthogonal to the space spanned by the 2nd, ..., nth rows of A. 


Ge- - 2 eal 
47, A= |-1 2 2h. 
2 —1 2 


A'A = 91 is diagonal: (A‘A);; = (column i of A)- (column j). 

49. (a) (1, —1,0) is in both planes. Normal vectors are perpendicular, planes still 
intersect! (b) Need three orthogonal:vectors to span the whole orthogonal com- 
plement in R°. (c) Lines can meet without being orthogonal. 


51. When AB = 0, the column space of B is contained in the nullspace of A. Therefore 
the dimension of C(B) < dimension of N(A). This means rank(B) < 4 — rank(A). 


Problem Set 3.2 


1. (a) (x + y)/2 => ./xy (arithmetic mean > geometric mean of x and y). 
(b) IIx+yl|l? < (lxl| + llyll)? means that @+y)™@+y) < |[x1]?-+2I xl yl] +Ilyll?. 
The left-hand side is x'x + 2xTy + y'y. After cancelling this is x’ y < ||x|||y|. 


3. p = (10/3, 10/3, 10/3); (5/9, 10/9, 10/9). 


1 
1 
5. cos@ = 1/./nso@ =arccos(1/./n); P = |: | [1/n --- 1/n] =allentries = 
1 
7. Choose b = (1,..., 1); equality if a; = --- = a, (then a is parallel to D). 
= aa‘aa™ a(a‘a)a‘™ aa’ 


~ aTaata (a™a)(aTa)” ata 


coer me 1 0 
11. (a) P = : A Pp=1]—-P, = = | (b) Pi + Py = fF if 
70 


10 
P, Py = f of The sum of the projections onto two perpendicular lines gives the 


vector itself. The projection onto one line and then a perpendicular line gives the 
zero vector. 
aay Gas. Gee 


13. Trace = Ty bt ata ne 


15. 


17. 


19. 


21. 


23. 


25. 


|| Ax ||? = (Ax)™(Ax) = xA™Ax, ||ATx|/? = (ATx)™(ATX) = xAATx. If ATA = 
AA’, then ||Ax|| = || A’x||. (These matrices are called normal.) 
(a) atb/ata = 5/3; p = (5/3, 5/3, 5/3); e = (—2/3, 1/3, 1/3) has e'a = 0. 
(b) atb/ata = —1; p = (1,3, 1) = bande = (0,0, 0). 

1 1 i 


1 1 5 i 1 3 1 
P=-/]1 1 1 = Prand Pib= = 5 -Py= 77 |3 9 3} and 
Slag: sh 5 i 3 4 
1 
Pob = {3}. 
1 
,f i -2 -2 ,f4 4-2 
ea ey ae ae | ee Oe ee 0 
91. 4 4 PND. aig | 4 
P,P) = zero matrix because a, | ap: 
P, + Po + P3 
1 1 -2 -2 1 4 4 -2 i 4 —2 4 
BIT oy Oe Peal se ee ae. Ah WS aa 
Pleo A. sale ee 2D. OE de ey A 
Since A is invertible, P = A(A™A)~!A? = AA7!(A‘)-1A™ = J: project onto all 


of R?. 


Problem Set 3.3 


1. 


e x = | 


. b=4,5,9 att = —1,0, 1; the best line is 6 + (5/2)t; p = (7/2, 6, 17/2). 


17. 


19. 


X = 2; E? = (10 — 3x)? + (5 — 4x)? is minimized; (4, —3)™(3, 4) = 0. 


jo 


;b-p= is perpendicular to both columns. 


Wir Whe 
WI Wis Wir 


WIN WIR Wir 


| 


i 4 =f 


. P=A(ATA) TAT = 11/2 1/2 1/2). 


Oe Sho 4 


. (a) P™ = (P™P)’ = P. Then P = P'P = P*. (b) P projects onto the space 


Z = {0}. 


. P+Q=T1, PQ = 0, transpose to OP = 0, so (P— Q)(P— Q) = P—-O0O-O+ 


Q=I. 


. Best line 61/35 — (36/35)t; p = (133/35, 95/35, 61/35, —11/35) from C + Dt. 
.W=(1-2PY =1—-4P4+4P*? =1 —4P +4P = 1, Two reflections give I. 


Projection onto x + y = 0 = Projection onto (—1, 1) = 


12. 1/2 
4/2 2% 


Projection matrix onto row space would be AT(AA‘)~'A if the rows were indepen- 
dent. 


21. 


23. 


25. 


27. 


29. 


31. 


33. 


35: 


37. 


39. 


41. 


Pe. - ey Pe 1 2 
Best line is | “4! ~1@2) 71) ae oe ee 
—G;Q2 Ayn} |X2 —dxb 1 


C = (ATA)“1ATb, AT = [1---1],b = (y1,.--5 Ym)! then 


ATA m 
1 -1 1 2 
C 
AS : sie See Die bse 0 : 
i ta : ae 
1 2 4 —5 
(a) ata =m,a'b = bi +---+b,. Therefore x is the mean of the b’s. (b) The 


variance is |le||? = 5~".,(b; -x)*. (c) p = (3,3,3), e = (—2, -1, 3), p'e =0. 
t i 1 


i tl 


(@ —x)\@—x)' = (ATA)!AT(D — Ax)(b — Ax)™]A(ATA)“!. For indepen- 
dent errors, substituting (b ~ Ax)(b — Ax)’ = oI gives the covariance ma- 
trix (A? A)~!ATo? A(A™A)~!. This simplifies to o?(A‘A)7!: neat formula for the 
covariance matrix. 


1 9 1 
-—b —X9 = —(by +--+ + dio). 
10 10 + 10” 0‘ it-:>+ Dio) 
b 0 1 
1 SA a 8 — : 3|.4.. 41 be. 
1 3 ID =| 2 . Change right side to p = 3/4 = H solves Ax = p. 
1 4 20 17 
ii al(e]\s 
Closest parabola: DY} ; 
I 3.9 F 8 
1 4 16 20 
4 8 26] [C 36 
ATAX =| 8 26 92] |D| = |112}. 
26 92 338] |\F 400 
(a) The best line is x = 1+ 4+, which goes through the center point G : b) = (2,9). 


(b) From the first equation: Cm + DS“ t; = 55 b;. Divide by m to get C+ Dt =b. 


= wii +++ wh 
5 C1214 /T) Asp = C01, 13/2195): 
b — ASw = (~1/21, 8/21, —4/21), (Atw) W? W (b — Aw) = 0. 


Problem Set 3.4 


1. 


(a) -4=C-—2D,-3 =C-—D,-1=C+D,0=C+2D._ (b) Best line 
—2 +f goes through all 4 points; E* = 0. (c) bis in the column space. 


13. 


15. 


A ie 


19, 
21. 
23. ao 
25. 
Zs 


29. 


. (—2uut)* —2uu’) = I—4uu?+4uutuu! =1;Q0 = 


. Projection on as: (—2/3, 1/3, —2/3); the sum is b itself; notice that aja[, a2aj , a3a3 


are projections onto three orthogonal directions. Their sum is projection onto the 
whole space and should be the identity. 


| 


Nike pe NI Vir 


| 
Nis iE Vie Vie 
| 


Nie NI NI Nie 


| 


NIE NIE Ve VIF 


© (igi Het #Xngn) Ogi +2 + nde) = xP +++ + XZ => bl? = dT = 


XP pee xe, 


. The combination closest to g3 is Og; + 0qp. 
11. 


Q is upper triangular: column 1 has qi; = £1; by orthogonality column 2 must be 
(0, £1, 0, ...); by orthogonality column 3 is (0, 0, +1, ...); and so on. 


Jo 0 1 00 1) f1 1 1 
A=|0 1 = {01 0 : : = OR. 
1 1 1 0 0 
1/3 2/3 —2/3 
a= 2/3|.9¢. = 11/3|.¢3 = 2/3 | 1s in the left nullspace; 
—2/3 2/3 1/3 


Rx = Q'Db gives ba a [2] = P| and X = cae 

C* — (gi C*) qz isc — (qie) a1 — (93¢) gz because g7qi = 0. 

By orthogonality, the closest functions are 0 sin2x = 0 and 0+ 0x = 0. 

= 1/2, a= 0, by = 2. 

The closest line is y = 1/3 (horizontal since (x, x”) = 0). 

(1/2, —1/V2, 0, 0), (1/6, 1/6, 2/6, 0), 

(~1/2V3, -1/2V3, 1/273, -1/V3). 

A=a=(l,~1,0,0);B=b—p= ($, 5,-1,0);C =c—py—pa = (4, 4,4, -1) 


3? 37 3? 
Notice the pattern in those orthogonal vectors A, B, C. Next, (1, 1, 1, 1)/4. 


31. (a) True. (b) True. Ox = xq) + x22. || Ox||? = x7 + x7 because gq]? q) = 0. 
Problem Set 3.5 
4 00 0 16 O O O 
00 0 4 0 146 #0 O 
2 4 me) 
oes 0 0 4 OV as 0 O 16 OO; sie 
0 4 0 0 0 O O 16 
3. The submatrix is F3. 


5. 


e* = —]| forx = (2k+ 1, e” =i for @ = 2kx + 7/2, k is integer. 


iL. 


13. 


45. 


17. 


19. 


21. 


23. 


. c= (1,0, 1,0). 
. (a) y = F times (1, 0, 0, ECAR 1 Ay: 


(b) c= (1, 1,1, 1)/4. 


1 1 4) 2 
= 0 Ceven = | 1 you ) = 0 

oe cats lO a0 "2 > for 
0 0 0 0 


co= (fot fit fot fs)/4,c1 = (fo-ifi— fa tifs)/4,c2 = (fo- fit f—f)/4, 
= (fo tif; — fo—ifs)/4; f odd means fy = 0, fy = 0, fa = —fi. Then cy = 0, 
C2 = 0, C3 = —c), $0 c is also odd, 


1 : oe 1 1 
7 f aD is Game 1 1 1 1 
te st ie —_ ..p7Hu 
oe 1 p) ; i eae | —] gs 
1 i? —j i 
1 1 
n= e2ti/6 and FP; = 1 re apis . 
eft /6 1 eftil3 e2ti/3 
0 1 O 
A = diag(1,i,i7,i7); P= |0 O 1] and P* leadtoA? -1=0. 
1 0 0 


Eigenvalues e) = 2—1-1=0, eo, =2-i-P =2, eg =2-(-l)-(-)) =4, 
e3=2-P—-1? =2., Check traceO0+2+4+2=8. 

The four components are (cy + c2) + (c; +3); then (cy ~— cr) + i(c; — c3); then 
(Cy + C2) — (C1 +3); then (co — C2) — i(cy — c3). These steps are the FFT! 


Problem Set 4.2 


13. 


. det (2A) = 8 anddet (—A) = (—1)* det A = 4 anddet (A) = j anddet (A7') = 2. 
. The row operations leave det A unchanged by Rule 5. Then multiplying a row 


by —1 (Rule 3) gives the row exchange rule: det B = —det A. 


. For the first matrix, two row exchanges will produce the identity matrix. The second 


matrix needs three row exchanges to reach /. 


. det A = 0 (singular); detU = 16; detU™ = 16; detU~' = 1/16; detM = 16 


(2 exchanges). 


. The new determinant is (1 — m£)(ad — bc). 
11. 


If | det Q{ is not 1 then det O” = (det Q)” would blow up or approach zero. But Q” 
remains an orthogonal matrix. So det Q must be 1 or —1. 

(a) Rule 3 (factoring —1 from each row) gives det (K T) = (—1)? det K. Then 
—det K = det K' = det K gives det K = 0. 


15. 


17. 


19. 


21. 


23. 
25. 


27. 
29. 
31. 


33. 


35. 


0 O00 1 
0 O 1 0 = 
(b) 0-100 has det = 1. 
-1 0 0 0 


Adding every column of A to the first column makes it a zero column, so det A = 0. 
If every row of A adds to 1, then every row of A — J adds to 0 and det(A — J) = 0. 


1 
But det A need not be 1: A = : | has det(A — J) = 0, but detA =0 4 1. 

2 
det (A) = 10, det(A7!) = 4, det(A — AD) = A? ~ 714+ 10 = 0 for A = 5 and 
a, 
Taking determinants gives (det C)(det D) = (—1)"(det D)(det C). For n even the 
reasoning fails (because (—1)” = +1) and the conclusion is wrong. 

d —b 
d—be 1 
HeeCAa\cedee | ed be cad be) =” ss 
BS Se sg a (ad— bey? ad—be 
ad—bc ad—be 

Determinant = 36 and determinant = 5. 
det (L) = 1, det(U) = —6, det(A) = —6, det(U~'L~') = —3, and 
det (U-'L-!A) = 1. 
Row 3 — row 2 = row 2 — row | so A is singular. 
A is rectangular so det (ATA) = (det A‘)(det A): these are not defined. 
The Hilbert determinants are 1,8 x 10-*,4.6 x 1074, 1.6 x 1077, 3.7 x 107”, 
5.4 x 1078 4.8 x 107, 2.7 x 10773, 9.7 x 10747, 2.2 x 107*3. Pivots are ratios 


of determinants, so the tenth pivot is near 10-*°/10~8 = 107!°: very small. 


The largest determinants of O—1 matrices forn = 1,2,..., are 1, 1,2,3,5, 9, 32, 
56, 144, 320, on the web at www.mathworld.wolfram.com/HadamardsMaximum 
DeterminantProblem.himl and also in the “On-Line Encyclopedia of Integer 
Sequences”: www.research.att.com. With —1s and 1s, the largest 4 by 4 determinant 
(see Hadamard in index) is 16. 

det + M) = 1+a+b+c+4d. Subtract row 4 from rows 1, 2, and 3. Then 
subtract a(row 1) + b(row 2) + c(row 3) from row 4. This leaves a triangular matrix 
with 1,1,1,and1+a+b+c+4d on its diagonal. 


Problem Set 4.3 


1. 


3. 


5. 


(a) @12d21d34a43 = 1; even, sodetA = 1. 

(b) b13bo2b3,b 44 = 18; odd, so det B = —18. 

(a) True (product rule). (b) False (all 1s). 

(c) False (det{1 1 0;0 1 1;1 0 1] = 2). 

The 1, 1 cofactor is F,_;. The 1, 2 cofactor has a 1 in column 1, with cofactor F,,_2. 
Multiply by (—1)'*? and also —1 to find F,, = Fy_1 + F,_2. So the determinants 
are Fibonacci numbers, except F,, is the usual F,,_1. 


11. 


13. 


15. 
17. 


19. 


21. 


23. 


25. 


27. 
29. 


31. 


33. 


3D; 


37. 
39. 


. Cofactor expansion: det = 4(3) — 4(1) + 4(—4) — 41) = —12. 
- (a) (2 —1)n! (eachtermn—1). (b) (1 + Ls ap Sore gaa) nt, 
(n — 1)! 


, 2! 
(c) 3 + 2n — 3). 


0 A|[zI 0] _[AB A Eo 0 0 A] _ 
-o |e a) =[0" thee a-13% Le 7] 


det iG | A = det(AB). Test A = [1 2], B = HI det is 4] aoe 
Si uk 2 — 

1 0A ; 
det(AB); A = a B= [1 2], det pq 0 = det(AB). Singular: 
rank(AB) < rank(A) <n <m. 
detA = 1+18+12—9—4—6 = 12, so rows are independent; det B = 0, so rows 
are dependent (row 1 + row 2 = row 3); detC = ~—1, C has independent rows. 
Each of the six terms in det A is zero; the rank is at most 2; column 2 has no pivot. 
Q1 182303244 has —, 14073032041 has +,sodetA = 0; 
detB=2-4-4-2-—1-4-4-1=48. 

(a) If ayy = ay = a33 = 0 then four terms are sure zeros. 

(b) Fifteen terms are zero. 

Some term @1,42g * ++ An» in the big formula is not zero! Move rows 1, 2, ..., into 
rows a, 6,..., @. Then these nonzero a’s will be on the main diagonal. 


4!/2 = 12 even permutations; det (J + Peyen) = 16 or 4 or 0 (16 comes from J + J). 


cere | 4 0 0 4 
C=|2 4 2| amdAC’= |0 4 0 = 41. Therefore A~! = 7C". 
12 3) . 0 0 4 


|B = |A,| — [Anil = (a+ 1) —n=l1. 


We must choose 1s from columns 2 and 1, columns 4 and 3, and so on. Therefore n 
must be even to have det A, + 0. The number of exchanges is in so C, = (-1)"”. 


S, = 3, Sy = 8, S3,= 21. The rule looks like every second number in Fibonacci’s 
sequence ..., 3,5, 8, 13, 21, 34, 55, ... so the guess is Sy = 55. The five nonzero 
terms in the big formula for 54 are (with 3s where Problem 39 has 2s) 81+ 1—9— 
9—9=55. 
Changing 3 to 2 in the corner reduces the determinant F>,42 by 1 times the cofactor 
of that corner entry. This cofactor is the determinant of 5,1 (one size smaller), 
which is F>,. Therefore changing 3 to 2 changes the determinant to Fy,42 — Fo, 
which is Poy,41. 
(a) Every det L = 1; detU, = det Ay = 2, 6, —6 for k = 1, 2, 3. 
(b) Pivots 5, &, 2. 
The six terms are correct. Row 1 — 2 row 2 + row 3 = 0, so the matrix is singular. 
The five nonzero terms in det A = 5 are 

MAD) -eEeVCVCVEC) = -OCDVQ2) = OQOCVED 

=O) -De 1): 


41. 


43. 


With aj, = 1, the —1, 2, —1 matrix has det = 1 and inverse (A7!);; =n +1 — 
max(i, /). 

Subtracting 1 from the n,n entry subtracts its cofactor C,,, from the determinant. 
That cofactor is C;,, = 1 (smaller Pascal matrix). Subtracting 1 from 1 leaves 0. 


Problem Set 4.4 
90) 40° 12 20° 10 2 
1. detA=20:;c'=|0 5 0 |;AC’=201:-A7=—|]0 5 0 
0 0 4 0io ogo 4 
3. (x, y) = (d/(ad — be), —c/(ad — be)); (x, y, 2) = (3, —1, —2). 


19, 


21. 


23. 


25. 


27. 


29. 


31. 


. (a) The area of that parallelogram is det Ms 


1 a so the triangle ABC has area 


5 4=2. (b) The triangle A’B’C’ has the same area; it is just moved to the origin. 


. The pivots of A are 2, 3, 6 from determinants 2, 6, 36; the pivots of B are 2, 3, 0. 
. (a) P? takes (1, 2, 3, 4, 5) to (3, 2,5, 4, 1). 


(b) P7! takes (1, 2, 3, 4, 5) to (3, 4,5, 2, 1). 


. The powers of P are all permutation matrices, so eventually one of those matrices 


must be repeated. If P’ is the same as P*, then P’~* = J. 


. (a) detA = 3, det B; = —6, det B, = 3, so x, = —6/3 = —2 and x2 = 3/3 = 1. 


(b) |A| = 4, |B] = 3, |Bo| = —2, |B] = 1.Sox = Fj and x, = ~} and x; = }. 


. (a) x1 = 2 andx. = =:nosolution (b) x, = 2 and x. = ©: undetermined. 
0 0 


0 0 


. If the first column in A is also the right-hand side b then det A = det B,. Both B, 


and B3 are singular, since a column is repeated. Therefore x, = |B,|/|A| = 1 and 
XQ = X3 = 0. 

If all cofactors = 0 (even in a single row or column), then det A = 0 (no inverse). 
A= i i has no zero cofactors but it is not invertible. 


If det A = 1 and we know the cofactors, then C? = A7! and also det A~! = 1. 
Since A is the inverse of A~', A must be the cofactor matrix for C. 


Knowing C, Problem 22 gives det A = (det C)*1 with n = 4. So we can construct 
A~' = C'/det A using the known cofactors. Invert to find A. 


(a) Cofactors Co; = C3; = C32 = 0. 
(b) Cy. = Cx, C3, = C13, C32 = Cx, make st symmetric. 


(a) Area 3 2! — 10. (b) Triangle area=5. (c) Triangle area = 5. 
1 4 


as Oe 
(a) Area 3/3 : ee (b) 5 + new triangle area $ - =5+7 = 12. 


The edges of the hypercube have length ./1 + 1+ 1-+1 = 2. The volume det H 
is 2* = 16. (H/2 has orthonormal columns. Then det(H/2) = 1 leads again to 
det i = 16,) 


33. 


39. 


41. 


43. 


45. 


X1y2—X2y; =rectangles A+ B+ D (not C). 


Areas from rectangle A = 2(triangle a) 
same bases _ rectangle B = 2(triangle b) 
same heights rectangle D = 2(triangle d). 


So triangles a + b +d = $(x1y2 ~ X21). 


Check an example with (a, b) = (3,2), (c,d) = (1,4), and area = 10. The line 
from (0, e) in step 3 has slope c/a and equation y = e+ cx/a. Step 3 works because 
(b, d) is on that line! d = e + cb/a is true, since ad — bc = area ae at step 2. 


. The n-dimensional cube has 2” corners, n2”-! edges, and 2n faces of dimension 


n — 1. The cube whose edges are the rows of 27 has volume 2". 


. J =r. The columns are orthogonal and their lengths are 1 andr. 


dr/ax dar/dy| _ cos @ sind |_ i 
a0/ax a0/dy| | (~sin@)/r (cos@)/r| Fr’ 

= (2,1, —1) gives a parallelogram, whose area is the length of a cross product: 
{PQ x PS|| = ||(—2, —2, —1)|| = 3. This comes also from a determinant ! The 
other four corners could be (0, 0, 0), (0, 0, 2), (1, 2, 2), (1, 1, 0). The volume of the 
tilted box is |det| = 1. 


x ¥y = 
det}3 2 1] =O = 7x —5y +z; plane contains the two vectors. 
12 3 


VISA has the five reversals VI, VS, VA, IA, SA. And AVIS has the two reversals 
Vi and VS. Since 5 — 2 is odd, VISA and AVIS have opposite parity. 


Problem Set 5.1 


11. 
13. 
ie 


. A=2andd = 3, trace = 5, determinant = 6. 
. 4 = —Sand A = —4; both A’s are reduced by 7, with unchanged eigenvectors. 
.A=3,i = 1,4 = 0, with eigenvectors (1, 0, 0), (2, —1, 0), (3, —2, 1); trace = 4, 


det = 0.A = 2,4 = 2, = —2, with eigenvectors (1, 1, 1), (0, 1,0), 7,0, —D); 
trace = 2, det = —8. 


. Ax = Ax gives (A — 7I)x = (A —7)x; Ax = Ax gives x = AA7!x, 80 Av'x = 


(1/A)x. 


. The coefficient of (—A)"~! in (Ay —A) +++ (Ag —A) i8 Ay H+ $A,. In det (A — AJ), 


a term that includes an off-diagonal a;; excludes both a;; — A and aj; — A. Sucha 
term doesn’t involve (—A)""!. So the coefficient of (—A)"~! in det (A — AJ) must 
come from the product down the main diagonal. That coefficient is aj; +---+@nn = 
At a eee Ae An: 

Transpose A — AJ: det (A — AI) = det(A — AJ)" = det (A? —AJ). 

The eigenvalues of A are 1, 2, 3, 7, 8, 9. 

rank(A) = 1,4 = 0,...,0,n (trace n); rank(C) = 2,4 = 0,...,n/2, —n/2 
(trace Q). 


17. 
19. 


21. 


23. 
25. 


27. 
29. 


31. 


33. 


35% 


37. 


The third row contains 6, 5, 4. 
A and A? and A® all have the same eigenvectors. The eigenvalues are 1 and 0.5 for 
A, 1 and 0.25 for A”, 1 and 0 for A®. Therefore A’ is halfway between A and A®. 
A, = 4 and A. = —1 (check trace and determinant) with x, = (1,2) and x2 = 
(2,1). A7! has the same eigenvectors as A, with eigenvalues 1/A,; = 1/4 and 
1/A. = —1. 
(a) Multiply Ax to see Ax which reveals A. (b) Solve (A — AJ)x = 0 to find x. 
(a) Pu = (uu")u = u(utu) = u,sod=1. (b) Pv = (uu")v = u(u'v) = 0, 
sox = 0. (c) xy = (-1, 1, 0, 0), x2 = (~3, 0, Ag 0), 3 C5; 0, 0, 1) are 
orthogonal to u, so they are eigenvectors of P with 2 = 0. 
3 —1=OgivesX} = land) = (—1 + i./3); the three eigenvalues are 1, 1, —1. 
(a) rank = 2. (b) det(B™B) =0. Not(c). (d) (B+/7)7'has(A+4+1)! = 
i 

’ 3? a 
a=0, b=9, c =O multiply 1,4, A? indet(A —AI) = 94 — A3: A = companion 
matrix. 


1 0 0 0 —| 
Hamilton). 


' of f ib 5 i Always A?= zero matrix if }= 0,0 (Cayley— 


Ax = CyAqxX1 tess #CpAnXn equals Bx = CyA,xX, +--+ + CyAnXp for all x. So 
A= B. 


a bj {1 at+b| _ 1]. sec 7 
E il f= etal -e+ Hi Ap = d — b to produce trace = a + d. 


39. We need A? = 1 but not A = 1 (to avoid J). With A, = e27/3 and A, = e7?"/3, the 
determinant will be 1,A2 = 1 and the trace is 4; +A2 = cos # +i sin = +cos me — 
isin . — —1. One matrix with this trace —1 and determinant 1 is A = E rl 

Problem Set 5.2 

jE el, SE ir 
Pode: a 1 —1;/0 0O —1} ’ 
A A OND aE 
0 0 QO ~2|j;0 O| |O —2 
1 
3. 4 = 0, 0, 3; the third column of S is a multiple of | 1] and the other columns are 
on the plane orthogonal to it. 1 
5. A; and A3 cannot be diagonalized. They have only one line of eigenvectors. 
FS: OV WV ew. ate (BAIS ON 
maa [i Bie iti i gives A = 1-1 Bula. 24'|| = 


1 3.51004 4 3.5100 _ 3 
4 5100 _ 4 510014 3 


11. 


13. 
15. 


17. 
19. 


21. 
23. 


2. 
p44 F 


29. 


31. 


33. 


35. 


37. 


39. 


41. 


. trace(A B) = trace(BA) = ag +bs+cr +dt. Then trace(AB — BA) = 0 (always). 


So AB — BA = / is impossible for matrices, since 7 does not have trace zero. 


111 1 0 O]... 
(a) True; detA = 240. (b) False; |}O 1 1]. (c) False; |0 1 O} is 
0 0 2 0 0 2 


diagonal! 


sof JR YE asf spose 
b-b lb db ke J-L Jb df” 
‘=f fo slo abo 


(a) False: don’t know A’s. (b) True. (c) True. (d) False: need eigenvectors 
of S! 

The columns of S are multiples of (2, 1) and (0, 1) in either order. Same for A7!. 
A and B have A, = 1 andAz = 1.4+Bhasd, = 1, A, = 3. Eigenvalues of A+ B 
are not equal to eigenvalues of A plus eigenvalues of B. 


(a) True. (b) False. (c) False (A might have 2 or 3 independent eigen- 
vectors). 


Whe Wile 
| 


8 3 9 4 10 5] 
A= E | (or other), A = is i A= B i} only eigenvectors are 
(c, —c). 
SA*S~' approaches zero if and only if every |A| < 1; B‘ > 0 from A = .9 and 
Ao 


9 0 3 23) 3 3 atl. 3 
ofS Bhs Bp el -onfh LoL 


B?° H = sum of those two. 


pe’ Or Tt <i). fee sea! 
~ 10 —1) |0 2) |0 -—1] ~~ |0 reann 
trace AB = (aq +bs)+ (er +dt) = (ga+rc)+(sb+td) = trace BA. Proof for 


diagonalizable case: the trace of SAS~! is the trace of (A.S~!)S$ = A, which is the 
sum of the r’s. 


The A’s form a subspace, since cA and A; + Az have the same S$. When S = J, the 
A’s give the subspace of diagonal matrices. Dimension 4. 


Two problems: The nullspace and column space can overlap, so x could be in both. 
There may not be r independent eigenvectors in the column space. 


A= f 3 has A? = F i} and A* — A — J = zero matrix confirms Cayley— 


Hamilton. 


43. 


45. 


By 5F, B has the same eigenvectors (1, 0) and (0, 1) as A, so B is also diagonal. The 


_|a 6b a 2b|  |0 0 _ ai 
equations AB — BA = EB aa = é a = | are —b = OQandc = 0: 
rank 2. 


A has A; = 1 and A, = .4 with x; = (1, 2) and x2 = (1, —1). A® has A, = 1 and 
do = 0 (same eigenvectors). A? has A; = 1 and Az = (.4)!, which is near zero. 
So A! is very near A. 


Problem Set 5.3 


1. 


11. 


13. 


45. 


17. 


The Fibonacci numbers start even, odd, odd. Then odd + odd = even. The next two 
are odd (from odd + even and even + odd). Then repeat odd + odd = even. 


2. 1 


bed Bo 2 


walt a= Bip at= (5 ah Boor. 


ee ve te F i =~ : ?: ‘ , al : “33) cootice 5-5, 
Bs 


1 O — Ar 1 1 QO do 1 A 


7 Ar Agi {ay 0 1 -a|fl] ff ----- 
= ere alk MILL Aa} LO} [CAR — a4) /@a — Ag) 


. Direct addition L;, + Ly4, gives Lo, ..., Ly a8 2, 1, 3, 4, 7, 11, 18, 29, 47, 76, 123. 


My calculator gives 41° = (1.618...)!° = 122.991..., which rounds off to 
Linco. 


7 1 
a 6 9 
. The Markov transition matrix is | 4 4 0]. Fractions 4, , 1 don’t move. 
1 1 
4 3 | 


(a) A = 0, (1,1,-2). (6) A=1and—0.2.  (c) limit (3, 4, 4) = eigenvector 
ford’ = 1. 


y Osasl 
@ gop. | 
_ [b/d—a) 1] fit 0 bia) -a 
(b) n= | 1 Ei i me 1 =i, 1 
2b Lab 
pea a ee eee ea gg cad yO 
b-~a+l1 b-—- aq ) 
241—-a) 1l—-—a-b Ae 
ee ee eee ep 
hoa bagel 
2b 3 
on el fas 
Quel wei GSe18 
20-4) not Markov. 
b—a+l 


The components of Ax add to x; + x2 +3 (each column adds to 1 and nobody is 
lost). The components of Ax add to A(x) + x2 +.x3). IfA 4 1, xy + x2 +3 must be 
zero. 


b “| is unstable for |~| > 1/2, and stable for |a| < 1/2. Neutral for @ = +1/2. 


19, 


21. 


23. 


25. 


0 0 2 E ck. @ 
A*=|0 0 O/ and A? =0.So(7—A)'=I1+A4A?=]0 1 1]. 

0 0 0 001 
If A is increased, then more goods are consumed in production and the expansion 


must be slower. Mathematically, Ax > tx remains true if A is increased; fmax gOeS Up. 


B sl=at alo }La eet=30 “i][o PEt a} 


2 p) 
R=SVAS7' = Hi Hl has R2 = A, /B would have A = /9 andd = /—I, so 


its trace is not real. Note that i : a can have ./—1 = i and —#, and real square 


27. 


29. 


root a 

aL TOF 
A = SA,S7' and B = SA,S7!. Diagonal matrices always give Ay Ay = AzAj. 
Then AB = BA, from SA; S~'SA2S~' = SA,A2S7! = SA2A\S = 
SA,S "SAS"! = BA. 
Bhasa = iand—i, so B*hasA* = 1and1;C has = (1£V3i)/2 = exp(+mi/3), 
so A? = —1 and —1. Then C? = —/ and Cc! = —C. 


Problem Set 5.4 


1. 


11. 


13. 


At Cbd Ble ag = 
-e =1+ar= |) A u(O) = 


Ay = —2 andA, = 0; x1 = (1, —1) and x, = (1, 1); 
e72t +1 ~—e4t +1] 
ged) ie hag y 


Atleoil 
Ces 


2t 


. u(t) = |_& Hes, as tf > 00, e > +00. 


; (a) eACtT) = SehG+D gol = SeAt eAT 5-1 —_ Sedt §—lgeAl §-} — ett AT. 


0 


1 0 —-1| ., 
jAta= i | gives 


Bis = 
(b) e -r+as[i 1 


Ave . |coSl —sin1 
e a 
sinl cos] 


Wc. _ fi -1 
he =r+3=[) 1 


| from Example 3 in the text, att = 1. This matrix is 


different from e4e?. 


4¢+3 
4 | 


~ (ajar = TST A? = Za5T Rea; > 0, unstable. (b) A; = V7, A. = —V7, 


Rea; > 0, unstable (c) A; = ali | Ar =kwB Re A; > 0, unstable 

(d) A; = 0, A2 = —2, neutrally stable. 

A, is unstable fort < 1, neutrally stable fort > 1. A, isunstable fort < 4, neutrally 
stable at t = 4, stable with real 1 for 4 < t < 5, and stable with complex 2 for 
t > 5. Az is unstable for all t > 0, because the trace is 21. 

(a) ui = cu2 — buz, ul, = —cu, +aUu3, U4 = bu; — au gives uu; +u,u2 +u4u3 = 
0. (b) Because e“’ is an orthogonal matrix, ||(t)||? = |je4*u(O)||* = |]u(O)|I? is 


15. 


17. 
19. 


21. 


23. 


25. 


27. 


29. 


31. 


33. 


35. 
37; 


39. 


constant. (c) A = Oand+(/a? + b? + c*)i. Skew-symmetric matrices have pure 
imaginary A’s. 


6 eee 1 
u(t) = 4 cos 2t 3] + cos V6t A 


Ax =AFx +)°x or (A —AF ~A71)x =0. 
Eigenvalues are real when (trace)? — 4det > 0 = —4(-a*- Bb? +c?) > 05 
a+b? >c’. 


u, =e ab Uy = e' | If u(0) = (5, —2), then u(t) = 3e* A + 2e! ra 


>. 7 k ; >} Then A = 5(5 + /41). 


y 4 5 
hd, =O and A, = 2. Now v(t) = 20 + 10e% — oo ast > ow. 
A= e ‘| has trace 6, det 9, A = 3 and 3, with only one independent eigenvector 
(1, 3). That gives y = ce*, y’ = 3e*'. Also te* solves y” = 6y’ — 9y. 
y(t) = cost starts at y(0) = 1 and y’(O) = 0. The vector equation has u = (y, y’) = 
(cost, —sint). 
Substituting u = ev gives ce’v = Ae“v — e“b, or (A—cl)v = b, orv = 
(A—cl)~'b = particular solution. If c is an eigenvalue, then A —c/ is not invertible: 
this v fails. 
de*' /dt =A+Att+5Z AP + iAP +--- = AU +Att GAP +I +.) = 
Ae. 
The solution at time t + T is also e4“+™ u(0). Thus e“’ times e47 equals e4¢+”. 
If A? = A then e*’ = 7+ At+ $A? +7AP4---=I+(e'-1)A 


ue Ol tered e’—1]_ je &—1 
~ |O 1 0 0 | {0 1 7 


_fi 1 _ fi a) fa of fo 3 a fe de*—e] _,, 
a= [i leap 0 k | f ne Sls eer ee 


41. (a) The inverse of e“’ ise~4’. (b) If Ax = Ax then e“’x = ex and e% $0. 
43. 4 = 2 and 5 with eigenvectors | and H .Then A = SAS7! = iS 
Problem Set 5.5 

1. (b) sum = 443%; product = 7+i. (c) 3+4) = 3-4; 1-i = 143; 


3. 


5 


13 + 4i| = 5; |1 —i| = V2. Both numbers lie outside the unit circle. 

¥ = 2-—i,x¥ =5,xy = -14+7i, 1/x = 2/5 — (1/5)i, x/y = 1/2 — (1/2)i; 
check that |xy| = /50 = |x|}y| and [1/x| = 1//5 = 1/|x]. 

(a) x? = r7e!?, x-! = (1/rje"”, % = re-"?; x“! = X gives |x|? = 1: on the unit 
circle. 


11. 


13. 


15. 


17. 
19. 
21. 
23. 


25. 


27. 


29, 


31. 


33. 


35. 


1 -i 1 iO 2 i -t 
.C={-i 0 | le —~i 1 O|,C® =Cbecause(AZA)¥ = AMA, 


Oo iI 


. (a) det AT = det A but det A" = det A. (b) A! = A gives detA = det A =real. 


Pik =O,d2= ha =|_ 5h n= |i MA ih = Lda =I, xy, = 
ve 7 ee ae PS) ec UY 
Wee y= WA pnetatne Pie a= mae i 


(a) u,v, w are orthogonal to each other. (b) The nullspace is spanned by wu; the 


left nullspace is the same as the nullspace; the row space is panic by v and w; 

the column space is the same as the‘row space. (c) x =v+ qt not unique, we 

can add any multiple of u tox. (d) Need b’u =0. (e) S7' = S'; SAS = 

diag(0, 1, 2). 

The dimension of S isn(n+1)/2, notn. Every symmetric matrix A is a combination 

of n projections, but the projections change as A changes. There is no basis of n 

fixed projection matrices, in the space S of symmetric matrices. 

(UV)BQUV) = V8UHUV = VEIV = 7. So UV is unitary. 

The third column of U can be (1, —2, i)/./6, multiplied by any number e’”. 

A has +1 or —1 in each diagonal entry; eight possibilities. 

Columns of Fourier matrix U are eigenvectors of P because 

PU =diag(1, w, w?, w*)U (and w =i). 

n’ steps for direct € times x; only nlogn steps for F and F~! by FFT (and n for A). 
2 0 bed 

AMA = | 0 2 I1+i and aA" = | 

l-i. 1-i 2 
(AHA)# = AH AHH — AMA again. 


1 a ; 
: | are Hermitian matrices. 


cA is still Hermitian for real c; A)" = —iA™ = ~—iA is skew-Hermitian. 
00) -1] 

P?=}1 0 O0}7, P=T7, P! = P™P = P; A =cube roots of 1 = 1, 
Oo 1 0 

e2ni/3 efi /3 


235 4 24+5+4 
C= {4 2 5) =24+5P+4P* hasA(C) = 2 + 5e2%i/3 4 4esi/3 
5 4 2 2 + 5e47i/3 4 4e8zi/3 4 


oT; 


39. 
41. 


43. 


45. 


47. 


49. 


yaelfitv3 -1+i) [1 o)ifi+v3 1: 
~ EL} iti 1473) (0 -1}L}/-1-i 144% 


V = V" givesreal A, unitary gives |A| = 1, then trace zero gives A = 1, —1. 


| with L? = 6+2/3. 


Don’t multiply e~** times e’* ; conjugate the first, then 1 Fg en ax ler? ea 0: 
R+iS =(R+iS)" = R™ —iS"; R is symmetric but S is skew-symmetric. 
a b+ 7 


with a7 +b? +c? =1. 
b—iIc 


[1] and [—1]; 
(I —2uu®)# = I —2uu®; UW —2uu®)? = I —4uu" + 4u(uBu)u® = I; the matrix 
uu" projects onto the line through wu. 

We are given A+iB = (A+iB)" = A? —iB*. Then A = A? and B = —B". 


1-—i 1—i}] |1 O} 1 |2+2% -2 “1 dened 
A= 4] 2 | 0 ‘lz bee | = SAS’. Real eigenvalues | and 4. 


Problem Set 5.6 


1. 


11. 


13. 


15. 
17. 


19. 


21. 


C = N!BN = N7!M!AMN = (MN)"!A(MN); only M“'IM = 1 is similar 
to I. 


. IfA1,..., A, are eigenvalues of A, thenA;+1,...,4,+1 are eigenvalues of A+. 


So A and A + J never have the same eigenvalues, and can’t be similar. 


. If B is invertible, then BA = B(AB)B™! is similar to AB. 
. The (3, 1) entry of M~'AM is gcos@ + hsin@, which is zero if tand = —g/h. 
. The coefficients are c} = 1, cp = 2, dy = 1, do = 1; check Mc = d. 


The reflection matrix with basis v; and v2 is A = ki of The basis V; and V2 (same 


reflection!) gives B = k | If M = i | then A= MBM™!, 


Or 2 t 
0 1 0 00 0 
(a) D=]0 O 2}. (b) D?=1]0 O O} = third derivative matrix. The third 
0 0 0 00 0 


derivatives of 1, x, and x? are zero, so D? = 0. (c)A = 0 (triple); only one 
independent eigenvector (1, 0, 0). 


; : 1 0; |O 1} jO O 0 1 
The eigenvalues are 1, 1, 1, —1. Eigenmatrices 0 a f | F i: E 4) 


(a) TT# =U AUUMAN(U!)#¥ = 1. (b) If T is triangular and unitary, then its 
diagonal entries (the eigenvalues) must have absolute value 1. Then all off-diagonal 
entries are zero because the columns are to be unit vectors. 

The 1, 1 entries of T?T = TT give |ty; |? = |ty,|? + |to[? + [tg |? sot = ty = 0. 
Comparing the 2, 2 entries of TT = TT# gives m3; = 0. So T must be diagonal. 
If N = UAU—, then NN¥ = UAU~!(U~')FABUE is equal to UA A#UE. 

This is the same as U AHMAU# = (UAU~!)#(UAU~!) = NEN. So N is normal. 


23. 


25. 


27. 


29. 


31. 


33. 


35. 


375 
39. 


41. 


43. 


lo of 


The eigenvalues of A(A — [)(A — 21) are 0, 0, 0. 


a2+bc ab+bd a b 1 0 0 0 
=, = ad { 
Always ie co ee | (a 7) ? a + (ad — bc) 01 0 i { 


M~'J3M = 0,so the last two inequalities are easy. Trying for MJ; = JM forces the 
first column of M tobe zero, $o M cannot be invertible. Cannot have J; = M7! J2.M. 


61 45 13 9 
10 _ 410 Aer erane) 
A =2 _ = fe =e Be a 
1 


0 0 1 0 0 1 — 1 0 P 0 
fi ih fe a ks 1 are similar; a i by itself and F 4 by 
itself. 


(a) (M~'AM)(M7!x) = M-!(Ax) = M~'0 = 0. (b) The nullspaces of A and 
of M~'AM have the same dimension. Different vectors and different bases. 
2 3 2 k k-1 oa =2 

fC” 20) 2g OF BE fet Oe ORO Th age Sie ee 
ee | ah P= |G eel ck ie 0 ety 
w(t) = (w(0) + tx (0) + $2? y(0) + 225z(0))e*. 
(a) Choose M; = reverse diagonal matrix to get Med; M; = Mi in each block 
(b) Mo has those blocks M; on its diagonal to get My 17M) = J". 
(c) AP = (M71)TJTM? is (M~1)™M)'JMoM™ = (MMoM")"'A(MMoM)), 
and A? is similar to’A. 
(a) True: One ‘has 1 = 0, the other doesn’t. (b) False. Diagonalize a nonsym- 
metric matrix and A is symmetric. (c) False: in 3 and Ol are similar. 


(d) True: All eigenvalues of A + J are increased by 1, thus different from the 
eigenvalues of A, 


Diagonals 6 by 6 and 4 by 4; AB has all the same eigenvalues as BA plus 6 — 4 
Zeros. 


Problem Set 6.7 


1. 


Ss 


ac—b?’ =2-—4 = -2 <0;x* +4xy +2y? = (x + 2y)* — 2y? (difference of 
squares). 

det (A — AI) = A* — (a +c)A+ac — b* = 0 gives A; = ((@t+c)+ 

\/(a — c)? + b?)/2 and Az = ((a +c) — \/ (a — c)* + 4b7)/2); A, > Ois asum of 
positive numbers; 12 > 0 because (a + c)? > (a — c)? + 4b? reduces to ac > b’. 
Better way: product A;A2 = ac — b’. 


. (a) Positive definite when —3 < b < 3. 


1b) Polis 0 1% eae 1 
(b) ; 4 = E b = f 4} (c) The minimum is “30 —- By 


1 b| |x ) oat hc] Ve 1 —b ss 
when E 5 PI ea HI which is P| er | (d) No minimum, let 


y—> w,x = —3y, then x — y approaches —oo. 


17. 


19. 


21. 


1-1 -l L sl SI] 


- (a) Ai = [—1 1 1}andA,=/-1 2 —2). 
= a a a 
(b) fy = (41 — X2 — x3)? = O when x; — x) — x3 = 0. 
i 0 0 
(c) fo = (xy — Xp — 3)? + (2 — 3x3)? +x3;L=}1-1 1 O}. 
1, =—3 1 


As b | = E 4 E i E ‘| the coefficients of the squares are the 


6 16 2 1); |0 4; ]0 1 


pivots in D, whereas the coefficients inside the squares are columns of L. 


. (a) Pivots are a and c — |b|?/a and det A = ac — |b|?. (b) Multiply |x2|? by 


(c—|b\?/a). (c) Now x# Ax isasumof squares. (d) det = —1 (indefinite) and 
det = +1 (positive definite). 


. a>1and (a — 1)(c — 1) > b’. This means that A — J is positive definite. 
» f(x,y) =x? +4xyt+9y? = («+ 2y)? +5ys f(x,y) =x? + Oxy +9y = 


(x +3y)’. 
x? AT Ax = (Ax)'(Ax) = length squared = 0 only if Ax = 0. Since A has indepen- 
dent columns, this only happens when x = 0. 


4 -4 8 

A= |-4 4 —8{ has only one pivot =4, rank = 1, eigenvalues 24, 0, 0, 
8 -—8 16 

det A = 0. 


ax? + 2bxy + cy” has a saddle point at (0, 0) if ac < b?. The matrix is indefinite 
(A < Oand a > 0). 


Problem Set 6.2 


1 


3. 
5. 


11. 


A is positive definite for a > 2. B is never positive definite: notice i | 


det A = —2b? — 3b” + 1 is negative at (and near) b = 3. 


If x'Ax > O and x' Bx > 0 for any x 4 0, then x'(A + B)x > 0; condition (1). 


La 1 3 


. Positive A’s because R is symmetric and /A > 0. R = ; ‘; Rs ie aI 


» IxTAy?? = |xTRIRy|? = |(Rx)"Ry|? < (by the ordinary Schwarz inequality) 


|| Rx |)? ||Ry |? = TR Rx)(yR Ry) = (x7 Ax)(yT Ay). 


A= 25 | has A = 1 and4, axes 1 ra and 5 ie along eigenvectors. 


. Negative definite matrices: (1) x'Ax <0 for all nonzero vectors x. (II) All 


the eigenvalues of A satisfy 4; < 0. (IID) det A; < 0, det A, > 0, det Az < 0. 


15. 


17. 


19. 


21. 


23. 


25. 


27. 


29. 
31. 


33. 


35. 


37. 
39. 


41. 


43. 


(IV) All the pivots (without row exchanges) satisfy d; < 0. (V) There is a matrix 
R with independent columns such that A = —R'R. 

False (Q must contain eigenvectors of A); True (same eigenvalues as A); True 
(OTAQO = Q-'AQ is similar to A); True (eigenvalues of e~4 are e~* > 0). 

Start from a;; = (row j of R')(column j of R) = length squared of column j of 
R. Then det A = (det R)* = (volume of the R parallelepiped)? < product of the 


_ lengths squared of all the columns of R. This product is @11@22 -- - Ann- 


2-1 O 2-1 -i1 
A= |-1 2 -—1] has pivots 2, 3, 4; A = |-1 2 ~1] is singular; 

0-1 2 —-1 -1 2 

1] fo 

A|{1l| = |0|. 
1 0 
x'Ax is not positive when (x), %2,x3) = (0,1,0) because of the zero on the 
diagonal. . 


(a) Positive definite requires positive determinant (also: all A > 0). (b) All pro- 
jection matrices except J are singular. (c) The diagonal entries of D are its eigen- 
values. (d) The negative definite matrix —J has det = +1 when n is even. 

Ay = 1/a? and Az = 1/b?, soa = 1/./A; and b = 1/./Xp. The ellipse 9x? +16y” = 
1 


1 has axes with half-lengths a = } and b = 3. 


_ fo, 3] _ [3 0] [3 1 f2 0 Pa ae: 
os ; 3 - i; | F Hf = i 4 ee fs ei 
ax? +2bxy +cy” =a(x+ by) 4 ae—b" y2- 2x? +8xy+10y? = (x +2y)? +2y?, 
xTAx = 2(x1 — $x2—- Ay)" “+ 5 (xp — x3)*; x’ Bx = (x; + x. + x3)’. B has one 
pivot. 


A and C™AC have Ay > 0, A. = 0.C(t) = 10+(1-1)0R,Q = fs By! 


R= r A C has one positive and one negative eigenvalue, but J has two positive 


eigenvalues. 


The pivots of A — $1 are 2.5, 5.9, —0.81, so one eigenvalue of A — +1 is negative. 
Then A has an eigenvalue smaller than 5. 


tank(CT AC) <rank A, but also rank(C™ AC) > rank((C)"!1C? ACC!) = rank A. 
No. If C is not square, C’ AC is not the same size matrix as A. 

6—4A/18 ~—3—A/18 
—3—1/18 6-—4)/18 


; 1} }1 
Eigenvectors 3} Hk 


Groups: orthogonal matrices; e’4 for all t; all matrices with det = 1. If A is positive 
definite, the group of all powers A* contains only positive definite matrices. 


54 
det = 0 gives A; = 54, A2 = =" 


Problem Set 6.3 


1. 


a, 


15. 


17, 


19. 


2° 2d 2 _ |l/v 17 _ | 4/717 
20 Ol has only o? = 85 with vy = [1/¥07), sou = | 7 Mad 
3475 


— /5 
: 1 has eigenvalues o; ear and of = : ey 


ATA=| 


1 Z 


Since A = A’, the eigenvectors of A’ A are the same as for A. Since Ay = ‘ (1—4/5) 
is negative, 0, = A, but a2 = —Az. The unit eigenvectors are the same as in 
Section 6.2 for A, except for the effect of this minus sign (because we need Av) = 


O22): 
pce: M/f1+ at a eee do/ V1 +23 
: 1/fi+r2 ‘ oe i a ae 


ATA = | 


. AAT= f 1 has of = 3 with uw; = ha and o; = 1 withu, = Gy ; 
1 1 0 1//6 1//2 
ATA=|1 2 1) haso? =3 with, = |2//6|,o7 =lwithy =|} O |, 
OL 1/V6 my vp) 
1//3 
and nullvector v3 = |—-1//3|. 
1/V3 
1 1 0 3 0 0 
Then F ‘ 1 = [uy Uy] i 1 i [vy D) v3)". 


. A= 12uv" has one singular value o, = 12. 

. Multiply UV" using columns (of UV) times rows (of ZV‘). 

. To make A singular, the smallest change sets its smallest singular value 02 to zero. 
. The singular values of A + J are not o; + 1. They come from eigenvalues of 


(A+ D'(A+ J). 
1 
: 100 0 1 
1 1 
Aves 4 ek 5a ° a 0 1 0|,Bt=]1 0 c= | : 
Tt}? 9 + { 
4 be ee Gis a 0 0 7 0 
i 
4 


A® is the right-inverse of A; B* is the left-inverse of B. 


1 = 
ATA= FE sl = f | i : | : 1} take square roots of 4 and 16 


6 10; 211 1] 10 16) |-1 
Li te OVP i a).. 13 0 eo a 
to obtain S= > | | F 4 & alae | and QO = AS* = 
1 
ae 3} 


(a) With independent columns, the row space is all of R"; check (A’A)Atb = A'b. 
(b) A™(AA™)~'b is in the row space because A! times any vector is in that space; 
now (ATA) Atb = ATAA'(AA')'b = A'Db. Both cases give ATAx* = A'D. 


21. 


23. 


Take A = E 0 and B = Ki i Then AB = f i From C+ in Problem 15 
£° 20 OF oe 

we have At = |? of Bo = _ | = (AB)*, and (AB)* 4 BTAt. 
2 2 


A=Q,5Q) = At = Q.0*Q1 = AAt = GQ; DD OF. Squaring gives 
(AAt)? = Q;2U*LUtOT = Q,LDt OT. So we have projections: (AA*)? = 
AA* = (AA*)! and similarly for A+ A. AAt+ and A+ A project onto the column 
space and row space of A. 


Problem Set 6.4 


1. 


P(x) = xf — x1X2 + x} — x2x3 +.x3 — 4x, — 4x3 has OP /dx, = 2x, — x — 4, 


(OP /Ox2 = —x; + 2x. — x3, and AP /dx3 = —x2 + 2x3 — 4. 
. OP, /ax =x+y =O0anddP,/dy=x+2y—3=O0givex = —3andy =3. P, 


has no minimum (let y > oo). Itis associated with the semidefinite matrix E of 


. Putx = (1,..., 1) in Rayleigh’s quotient (the denominator becomes n). Since R(x) 


is always between A, and A, we get nd, < x'Ax = sum of all aj; < Nan. 


. Since x! Bx > 0 for all nonzero vectors x, x7(A + B)x will be larger than x7 Ax. 


So the Rayleigh quotient is larger for A + B (in fact all m eigenvalues are increased). 


. Since x’Bx > 0, the Rayleigh quotient for A + B is larger than the quotient for A. 
. The smallest eigenvalues in Ax = Ax and Ax = AMx are 4 and (3 — /3)/4. 
. (a) Aj = mins,[Max, ins, R(x)] > O means that every S; contains a vector x 


; — AC AC 7A 
with R(x) > 0. (b) y = C7'x gives quotient R(y) = Bacteria 


yly x7x 
R(x) > 0. 


. The extreme subspace S, is spanned by the eigenvectors x; and x3. 
17. 


If Cx = C(A~'b) equals d then CA~'b — d is zero in the correction term in 
equation (5). 


Problem Set 6.5 


1. 


3 


2. 1. “O13 /16 1/2 
Ay =bis4|-1 2 —1{ [4/16] = b = |1/2|. The linear finite element 
Ob. “2 tS /16 1/2. 
U=s3Vit ZVt V3 equals the exact u = =, <, = at the nodes x = i 5 3, 
2-1 O 1 2 
A33 = 3,b; = §.Then A =3]-1 2 —1},b = = |2], Ay = b gives 
| | ee etd 


5. Integrate by parts: He — V,'V; dx = fj ViVi dx — LAA = fe ViVjdx = 


same A;;. 


7A=4,M= i Their ratio 12 (Rayleigh quotient on the subspace of multiples of 


V (x)) is larger than the true eigenvalue A = zr”. 


9. The mass matrix M is h/6 times the 1, 4, 1 tridiagonal matrix. 
Problem Set 7.2 
1. If Q is orthogonal, its norm is ||Q|| = max ||Qx||/||x|| = 1 because Q preserves 
length: ||Qx|| = ||x|| for every x. Also Q-' is orthogonal and has norm one, so 
c(Q) = 1. 
3. || ABx|| < ||Al||| Bx], by the definition of the norm of A, and then || Bx|| < || B]|||x||. 


11. 


13. 
15, 


17. 


19. 


21. 


23. 


25. 


Dividing by ||x|| and maximizing, ||AB|| < ||Al|||B||. The same is true for the 
inverse, ||B~!A-!|| < ||B7'||||A7'I]; c(AB) < c(A)c(B) by multiplying these 
inequalities. 


. In the definition || Aj] = max || Ax||/||x||, choose x to be the particular eigenvector 


in question; || Ax || = |A|||x||, so the ratio is |A| and maximum ratio is at least |A|. 


. ATA and AAT have the same eigenvalues, since ATAx = Ax gives AAT(Ax) = 


A(A‘Ax) = (Ax). Equality of the largest eigenvalues means ||A|| = ||A‘||. 


0 0 1 0 

and Amax(AB) > Amax(A)Amax(B). SO Amax(A) is not a norm. 
(a) Yes, c(A) = ||Al| ||A7!|] = c(A74), since (A7!)~! is A again. (b) A7'b=x 

||55]| 1, 18xl] 4... 8xll — 1 {65 
leads to ——- < ||A]| ||A” ||_—-. This 1s > 

joy ae Il =e [ol 
| Al] = 2andc = 1; ||A|| = /2andc is infinite (singular !); ||A|] = /2andc = 1. 
If Amax <= Amin = 1, then all A; = 1 and A = SIS~! = J. The only matrices with 
| A|| = ||A7!|| = 1 are orthogonal matrices, because A‘A has to be /. 
The residual b — Ay = (1077, 0) is much smaller than b — Az = (.0013, .0016). 
But z is much closer to the solution than y. 


eo F iF B= : A Amax(A + B) > Amax(A) + Amax(B) (since 1 > 0 +0), 


x? +--+ x? is not smaller than max(x7) = ((|x||o0)* and not larger than 
(lx1] + - ++ + |xp])?, which is (|x ||1)?. Certainly x? + --- + x? <n max(x?), so 
\|x || < \/n||x loo. Choose y = (sign x;, sign x2,..., signx,) to get x+y = ||x||,. By 
Schwarz, this is at most ||x|| || y|| = ./||x ||. Choose x = (1, 1,..., 1) for maximum 
ratios /n. 
9 36 30 

The exact inverse of the 3 by 3 Hilbert matrix is A7' = |-36 192 —180]. 

30 —180 180 
The largest ||x|| = || A~'D|| is 1/Amin; the largest error is 107!°/Ajin. 


1 0 22, Zz. 2 : 0 1 
Exchange E 4 to i A —> E | = U with P = f i and 


10 22s 2. 2 0 2 iae® 
z=|5 qh4s fi o 1] fo -1 1} > fo 2 O| > 

02 0 ae ae 0 -1 1 
2.2 0 0 1 0 1 0 0 
0 2 0} =U.ThenPA=LUwithP= |0 0 1] andL= | 0 1 0 
O:70< A 1 0 0 5 -5 1 

Problem Set 7.3 
fais 1 ite 2 paar 5 i re 1| normalized to 
PET heeds eda eats 13)" ~ 5 —1] unit vector. 

3. uy / AK = C1x1 +CoX2(A2/Aq)E +++ + + epXn(An/A1)* > c1x1 if all ratios |A;/A;| <1. 
The largest ratio controls, when k is large. A = k 0 has |A2| = |A;| and no 
convergence. . 

Die atayt 
5. Pitas ese n9) OE ie yy mata) = Hy is 
(x —y)"@ — y) 
cs Hy, 
a 1 0 0 1 -5 0 
10S E 1 0 <} —3| =U andthnU“AU=|-5 & #I. 
9 4 3 12 16 
5 5 2% 25 
9 cos@ sin@| _ OR= cos@ —sin@| |1 cosé@ sind 
‘|sind Of] ~~~ J|sing cos@}| {0 —sin?@ | 
_ feQ+s?) —s3 
Then RQ = | 2 ae 
11. Assume that (Qo--+ Qp-1)(Re_1 +++ Ro) is the OR factorization of A* 
(certainly true if k = 1). By construction, Aji; = R~Qz, 80 Re = AgsiQ, = 
(Q) --» Q}AQo--+ Ox) Q}. Postmultiplying by (R;,_1 --- Ro), the assumption gives 
R,---Ro = Q; --- Qj A**. After moving the Q’s to the left-hand side, this is the 
required result for A**". 
13. A has eigenvalues 4 and 2. Put one unit eigenvector in row 1 of P: it is either 
1/1 ~1 1 {2 - oe ee 1. (4 - 
15. P;;A uses 4n multiplications (2 for each entry in rows i and /). By factoring out 
cos@, the entries 1 and + tan@ need only 2n multiplications, which leads to én? 
for PR. 
Problem Set 7.4 
0 5 0 
1, D'(-L—U) = |% 0 4), eigenvalues uw = 0, £1//2; (D+ L)"(-U) = 
; ee 


0 
0 


, eigenvalues 0, 0, 1/2; @opt = 4-2/2, reducing Amax to 3—2/2 © 0.2. 


Colm Bik pole 
Ble Nie ©} 


11. 


13. 


15. 
Li. 


19. 


. Axp = (2 —2coskmrh)x,; Ix, = + (sin 2kah, sin3krh + sinkih,...) = 


1 
(cos karh)x;,. Forh = rt A has eigenvalues 2—2cos = = 2— V/2,2-—cos . a 
3 
2 — cos = =2+ 72. 


go i ft 
ae 

Z 1 

ne DU(L +U)=-—{|0 0 4 ; the three circles have radius r; = 37 2 = 7Z 
3 3 0 


m= 5" Their centers are at zero, so all |A;| < 4/5 < 1. 


1/2 
.-DTV(L+U)= » ca has uw =+ (~) --(D+L)'U = 


c/d 0 ad 


0 -b/a|, _ ; 
FE ee A = 0, be/ad; Annas does equal Maes 


. If Ax = Ax, then (J — A)x = (1 — A)x. Real eigenvalues of B = J — A have 


{1 — A| < 1, provided that A is between 0 and 2. 

Always ||AB|} < |[Alj||B||. Choose A = B to find ||B?|| < ||Bl]?. Then choose 
A = B* to find ||B?|| < ||B?||||Bi]| < ||B\>. Continue (or use induction). Since 
|| B || = max |A(B)|, it is no surprise that ||B|| < 1 gives convergence. 


Es ,fo i_. ‘ ; = 0 
Jacobi has $ T= 3 1 0 with |A|max = 3- Gauss—Seidel has S“'T = 0 


Oi Wire 
[ we | 


with |Almax = 3 = (lAlmax for Jacobi)’. 
Successive overrelaxation (SOR) in MATLAB. 


The maximum row sums give all |A| < .9 and |A| < 4. The circles around diagonal 
entries give tighter bounds. First A: The circle |A —.2] < .7 contains the other circles 
|A — .3| < .5 and |A — .i| < .6 and all three eigenvalues. Second A: The circle 
|A — 2| <2 contains the circle |A — 2| < 1 and all three eigenvalues 2 + 2, 2, and 
2— /2. 

ry = b—ayAb = b— (b'b/b" Ab) Ab is orthogonal to ro = b: the residuals 
r = b— Ax are orthogonal at each step. To show that p; is orthogonal to App = Ab, 
simplify p; to cP): P; = ||Ab\|?b — (b"Ab) Ab and c = b'b/(b™ Ab)’. Certainly 
(Ab)' P, = 0, because AT = A. (That simplification put a; into p) = b — a,Ab+ 
(b'b — 20,b" Ab + a || Ab||*)b/b"b. For a good discussion see Numerical Linear 
Algebra by Trefethen and Bau.) 


Problem Set 3.1 


. The corners are at (0, 6), (2, 2), (6, 0); see Figure 8.3. 
. The constraints give 3(2x + Sy) + 2(—3x + 8y) < 9 — 10, or 3ly < —1. Can’t 


have y > 0. 


. x > 0, y => 0, with added constraint that x + y < 0 admits only the point (0, 0). 
. x (5% bonds) = z (9% bonds) = 20,000 and y (6% bonds) = 60,000. 


9. 


The cost to be minimized is 1000x + 2000y + 3000z + 1500u + 3000v + 3700w. 
The amounts x, y, z to Chicago and u, v, w to New England satisfy x +u = 
1,000,000; y + v = 1,000,000; z+ w = 1,000,000; x + y+ z = 800,000; 
u+uv+w = 2,200,000. 


Problem Set 8.2 


1. 


9. 
11. 


At present x4 = 4 and xs = 2 are in the basis, and the cost is zero. The entering 
variable should be x3, to reduce the cost. The leaving variable should be x5, since 
2/1 is less than 4/1. With x3 and x, in the basis, the constraints give x3 = 2, x4 = 2, 
and the cost is now xj + x» — x3 = —2. 


. The “reduced costs” arer = [1 1], so change is not good and the corner is optimal. 
. AtP,r=[~5 3]; thenat Q,r = [3 —4]; R is optimal because r > 0. 
. For a maximum problem the stopping test becomes r < 0. If this fails, and the ith 


component is the largest, then that column of N enters the basis; the rule 8C for the 
vector leaving the basis is the same. 


BE = B{.--v---]=[---u---], since Bu = u. So E is the correct matrix. 
If Ax = 0, then Px = x — A™(AA")“!Ax =x. 


Problem Set 8.3 


1, 


11. 


13. 


15. 


17. 


Maximize 4y,; + 1ly2, with y; > 0, yo > 0, 2y; + yo < 1, 3y2 < 1; the primal has 


xf = 2, x} = 3, the dual has y* = i, yz = 5, cost = 5. 


. The dual maximizes yb, with y > c. Therefore x = b and y = c are feasible, and 


give the same value cb for the cost in the primal and dual; by 8F they must be optimal. 
If b, < 0, then the optimal x* is changed to (0, b2,..., bn) and y* = (0, c2,..., Cn). 


.b=[0 if’ andc=[-1 0]. 
. Since cx = 3 = yb, x and y are optimal by 8F. 
.x*=[1 Of’, y* =[1 0], with y*b = 1 = cx*. The second inequalities in both 


Ax* > band y*A < c are strict, so the second components of y* and x” are zero. 
(a) xf = 0, x7 = 1, x¥ = 0,c'x = 3. (b) It is the first quadrant with the 
tetrahedron in the corner cut off. (c) Maximize y,, subject to y, > 0, y; < 5, 
wu <3,n<4 yf =3. 


Herec=[1 1 1]withA= k : | . No constraint x > 0 so the dual will 
have equality yA = c (or Aly = c'). That gives 2y, = 1 and y; = 1 and y. = 2 
and no feasible solution. So the primal must have oo as maximum: x; = ~—WN and 
X2 = 2N and x3 = 0 give Cost = x; + x2 + x3 = N (arbitrarily large). 
100-1 0 0 100 -1 
The columns of }O0 1 0 0 -l Olor/O 1 0 —1}. 
0 O.7 0 oO -l 00 1 -1 


Take y = [1 —1]; then yA > 0, yb <0. 


Problem Set 8.4 


1. 


11. 


13. 


15. 


The maximal flow is 13, with the minimal cut separating node 6 from the other 
nodes. 


. Increasing the capacity of pipes from node 4 to node 6 or node 4 to node 5 will 


produce the largest increase in the maximal flow. The maximal flow increases from 
8 to 9. 


. Assign capacities = 1 to all edges. The maximum number of disjoint paths from s 


to t then equals the maximum flow. The minimum number of edges whose removal 
disconnects s from ¢ is the minimum cut. Then max flow = min cut. 


. Rows 1, 4, and 5 violate Hall’s condition; the 3 by 3 submatrix coming from rows 1, 


4,5, and columns 1, 2,5 has3 +3 > 5. 


. (a) The matrix has 2n 1s which cannot be covered by less than n lines because 


each line covers exactly two Is. It takes n lines; there must be a complete matching. 


1 2 > fa 
100 0 1 
(b) }1 O O O 1). The Is can be covered with four lines; five marriages are 
1000 1 
Is “a, iid ab 


not possible. 

If each m + 1 marries the only acceptable man m, there is no one for #1 to marry 
(even though all are acceptable to #1). 

Algorithm 1 gives 1-3, 3-2, 2-5, 2-4, 4-6, and algorithm 2 gives 2-5, 4-6, 24, 
3-2, 1-3. These are equal-length shortest spanning trees. 

(a) Rows 1, 3,5 only have 1s in columns 2 and 4. (b) Columns 1, 3, 5 (in rows 2, 


4). (c) Zero submatrix from rows 1, 3,5 and columns 1,3,5. (d) Rows 2,4 and 
columns 2, 4 cover all 1s. 


Problem Set 8.5 


1. 


—10x,+70(1 ~x1) = 10x; -10(1 —x1), or xy = 3, x2 = 2; -10y; +10(1—y1) = 
70y, — 10(1 — y1), or y; = 3, yo = §; average payoff yAx = 6. 


. If X chooses column j, Y will choose its smallest entry a;; (in row i). X will not 


move, since this is the largest entry in that row. In Problem 2, aj. = 2 was an 
equilibrium of this kind. If we exchange the 2 and 4 below it, no entry has this 
property, and mixed strategies are required. 


. The best strategy for X combines the two lines to produce a horizontal line, guaran- 


teeing this height of 7/3. The combination is 3(3y + 2(1 — y)) +4(y +3. —y)) = 
1 


7/3, so X chooses the columns with frequencies 2, 0, qi 


. For columns, we want xja + (1 — x1)b = x;c+ (1 — x;)d = u, so 


x\(a—b—c+d)=d — b. For rows, ya+ (1 — yijce = yb + — yj)d =v 
exchanges b and c. Compare u with v: 


(a — b)(d — b) ad — be 


u=x1(a—b)+b = ———-_—__ + b = —--—-_—-. = same afterb > c = v. 


a-b—-ct+d TEan-bactd 


to yj + y2 = 1, the minimum of the larger y is 4 =. Notice A = T. 
1 


11, Ax* = [5 1" and yAx* = Sy + Sy = ; for all strategies of Y; y*A 
[$ $ -1 1] and y*Ax = 1x1 + 4x2 — x3 — x4, which cannot exceed 4; 
between is y*Ax* = 3 

13. Value 0 (fair game). X chooses 2 or 3, y chooses odd or even: x* = y* = (5, 3 


Problem Set A 


1. (a) Largest dim (S 9 T) =7whenSc T. 
(c) Smallest dim (S + T) = 8 when Sc T. 
(d) Largest dim (S + T) = 13 (all of R!°). 


(b) Smallest dim(S M T) = 2. 


. The inner maximum is the larger of y; and y2; : concentrates on that one. Subject 


in 


). 


ay ayn 43 4 aj, a2 O 0 
3. V + W and VN W contain @21 422 423, FAT ang 0 a2 a 0 . 
432 433-34 0 O @3 a4 
: 0 0 443° ~A44 0 0 0 aA44 
dim (V + W) = 13 and dim (V 1 W) = 7; add to get 20 = dim V + dim W. 
5. The lines through (1, 1, 1) and (1, 1, 2) have VN W = {0}. 
7. One basis for V + W is v1, v2, wi; dim (VM W) = 1 with basis (0, 1, —1, 0). 
9, The intersection of column spaces is the line through y = (6, 3, 6): 
1 5 1 ame) 7 1 5 3 0 
y= }3 0 |= 0 1 [| matches [4 Blx= 33° -0 0: 1 _9|= 0. 
2 4 0.2 240 2 3 
The column spaces have dimension 2. Their sum and intersection give 3+ 1 = 242. 
1 1 1 1 
_{|h Fol. [1 = 1 -l 
a 2 = =H is He ste 
1-1 ~—1 1 
13. Asp = (Aip @1 @1)+ UT @Aitp @1I)+ TU @I @ Ap). 
Problem Set 8 
7 0 1 O 
Pr (A is diagonalizable); J = |0 O 0] (eigenvectors (1, 0,0) and 
0 ‘ 
0 0 0 
(2,10). 
1 ¢ 2t 
3. c®%'= 10 1 O| =14Brsince B? = 0. Alsoe” =] +4 Jt. 
0 0 0 
1 0 0 0 
5. J= |0 4 0} (distinct eigenvalues); J = (B has 4. = 0, 0 but rank 1). 
0 0 6 ° 


